NAVAL 

POSTGRADUATE 

SCHOOL 

MONTEREY,  CALIFORNIA 


THESIS 


AN  EXPLORATORY  STUDY  OF  THE  UNITED  STATES 
NAVAL  ACADEMY  ENGINEERING  CURRICULUM 

by 

Michael  T.  O'Reilly 
June  2007 

Thesis  Co-Advisors:  Susan  P.  Hocevar 

Linda  D.  Mallory 


Approved  for  Public  release;  distribution  is  unlimited. 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


REPORT  DOCUMENTATION  PAGE 


Form  Approved  OMB  No.  0704- 

0188 _ 

Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per 
response,  including  the  time  for  reviewing  instruction,  searching  existing  data  sources,  gathering 
and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send 
comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  headquarters  Services,  Directorate 
for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA 
22202-4302,  and  to  the  Office  of  Management  and  Budget,  Paperwork  Reduction  Project  (0704-0188) 
Washington  DC  20503. 

I.  AGENCY  USE  ONLY  (Leave  blank)  2.  REPORT  DATE  3.  REPORT  TYPE  AND  DATES  COVERED 

June  2007  Master's  Thesis 

4.  TITLE  AND  SUBTITLE  An  Exploratory  Study  of  the  5.  FUNDING  NUMBERS 

United  States  Naval  Academy  Engineering  Curriculum 

6.  AUTHOR ( S )  Michael  T.  O'Reilly  _ 

7.  PERFORMING  ORGANIZATION  NAME ( S )  AND  ADDRESS (ES)  8 .  PERFORMING  ORGANIZATION 

Naval  Postgraduate  School  REPORT  NUMBER 

Monterey,  CA  93943-5000 

9.  SPONSORING  /MONITORING  AGENCY  NAME ( S )  AND  ADDRESS (ES)  10.  SPONSORING/MONITORING 
N/A  AGENCY  REPORT  NUMBER 

II.  SUPPLEMENTARY  NOTES  The  views  expressed  in  this  thesis  are  those  of  the  author  and 
do  not  reflect  the  official  policy  or  position  of  the  Department  of  Defense  or  the  U.S. 
Government . 

12a.  DISTRIBUTION  /  AVAILABILITY  STATEMENT  12b.  DISTRIBUTION  CODE 

Approved  for  public  release;  distribution  is  unlimited. 

13.  ABSTRACT  (maximum  200  words) 

The  purpose  of  this  study  is  to  quantitatively  assess  Naval  Academy 
graduates'  perceptions  of  two  aspects  of  their  undergraduate  education  as 
engineering  majors:  1)  the  extent  to  which  their  undergraduate  education  is 

relevant  to  their  current  profession,  and  2)  their  level  of  preparedness  as  a 
result  of  their  engineering  education.  The  Accreditation  Board  of  Engineering 
and  Technology  (ABET)  identifies  eleven  'student  learning  outcomes'  that  are 
utilized  as  the  basis  for  assessing  relevance  and  preparedness.  Baseline  data 
is  established  for  engineering  graduates  of  the  Naval  Academy  between  the  years 
of  1985  -  2005.  In  addition  to  the  general  analysis,  graduates  are  grouped  for 
comparison  and  analysis  according  to  status  (civilian  and  military),  job  type 
(technical  and  non-technical )  and  according  to  their  particular  undergraduate 
majors.  The  results  indicate  high  levels  of  both  applicability  and  preparedness 
for  most  of  the  eleven  skills.  Recommendations  for  future  engineering  program 
improvements  are  offered. 

14 .  SUBJECT  TERMS  15.  NUMBER  OF 

Naval  Academy;  Engineering;  Accreditation;  ABET;  Education  PAGES 

_ 105 _ 

16.  PRICE  CODE 

17 .  SECURITY  18  .  SECURITY  19.  SECURITY  20 .  LIMITATION  OF 

CLASSIFICATION  OF  CLASSIFICATION  OF  THIS  CLASSIFICATION  OF  ABSTRACT 

REPORT  PAGE  ABSTRACT 

Unclassified  Unclassified  Unclassified  UL 

NSN  7540-01-280-5500  Standard  Form  298  (Rev.  2-89) 

Prescribed  by  ANSI  Std.  239-18 


1 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


11 


Approved  for  public  release;  distribution  is  unlimited. 


AN  EXPLORATORY  STUDY  OF  THE  UNITED  STATES  NAVAL  ACADEMY 

ENGINEERING  CURRICULUM 

Michael  T.  O'Reilly 
Lieutenant,  United  States  Navy 
B.S.,  United  States  Naval  Academy,  2001 

Submitted  in  partial  fulfillment  of  the 
requirements  for  the  degree  of 


MASTER  OF  SCIENCE  IN  LEADERSHIP  AND  HUMAN  RESOURCE 

DEVELOPMENT 


from  the 


NAVAL  POSTGRADUATE  SCHOOL 
June  2007 


Author:  Michael  T.  O'Reilly 


Approved  by:  Dr.  Susan  P.  Hocevar 

Co-Advisor 


Dr.  Linda  D.  Mallory 
Co-Advisor 


Dr.  Robert  N.  Beck 

Dean,  Graduate  School  of  Business  and  Public 
Policy 


in 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


IV 


ABSTRACT 


The  purpose  of  this  study  is  to  quantitatively  assess 
Naval  Academy  graduates'  perceptions  of  two  aspects  of  their 
undergraduate  education  as  engineering  majors:  1)  the 
extent  to  which  their  undergraduate  education  is  relevant  to 
their  current  profession,  and  2)  their  level  of  preparedness 
as  a  result  of  their  engineering  education.  The 
Accreditation  Board  of  Engineering  and  Technology  (ABET) 
identifies  eleven  'student  learning  outcomes'  that  are 
utilized  as  the  basis  for  assessing  relevance  and 
preparedness.  Baseline  data  is  established  for  engineering 
graduates  of  the  Naval  Academy  between  the  years  of  1985  - 
2005.  In  addition  to  the  general  analysis,  graduates  are 
grouped  for  comparison  and  analysis  according  to  status 
(civilian  and  military),  job  type  (technical  and  non¬ 
technical)  and  according  to  their  particular  undergraduate 
majors.  The  results  indicate  high  levels  of  both 
applicability  and  preparedness  for  most  of  the  eleven 
skills.  Recommendations  for  future  engineering  program 
improvements  are  offered. 
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I. 


INTRODUCTION 


A.  BACKGROUND 

The  primary  purpose  of  the  United  States  Naval  Academy 
is  to  provide  graduates  who  then  serve  as  junior  officers  in 
the  U.S.  Navy  and  Marine  Corps.  A  significant  emphasis  of 
the  development  of  midshipmen  is  on  their  undergraduate 
education.  Graduates  are  awarded  both  a  commission  into  the 
Navy  or  Marine  Corps  and  a  Bachelor  of  Science  degree.  On 
average,  more  than  a  third  of  each  graduating  class  will 
have  majored  in  one  of  eight  different  engineering  fields. 

Following  graduation,  graduates  serve  for  a  minimum  of 
five  years  on  active  duty  in  one  of  many  different  warfare 
communities  within  the  Navy  or  Marine  Corps.  Many  remain  on 
active  duty  well  past  this  minimum  commitment,  and  others 
move  on  to  careers  within  the  civilian  workforce.  Regardless 
of  where  the  Naval  Academy  graduates  end  up  going,  or  what 
they  end  up  doing,  it  is  important  to  understand  how  their 
undergraduate  education  has  impacted  their  careers  since 
graduation . 

B .  PURPOSE 

There  are  several  purposes  to  this  study.  The  primary 
purpose  is  to  determine  the  extent  to  which  Naval  Academy 
graduates  who  majored  in  engineering  believe  their 
undergraduate  education  to  be  relevant  to  their  current 
profession.  This  portion  of  the  study  uses  the  Accreditation 
Board  of  Engineering  and  Technology  (ABET)  'student  learning 
outcomes, '  which  are  a  set  of  11  engineering  related  skills 
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or  abilities  (ABET,  2005,  p.  1)  as  the  basis  for  assessing 
relevance.  The  primary  goals  are:  (a)  to  determine  the 
extent  to  which  Naval  Academy  graduates  believe  these  skills 
are  relevant  to  their  current  profession,  and  (b)  to  examine 
how  well  graduates  believe  they  were  prepared  in  these  areas 
through  their  undergraduate  education. 

Another  purpose  of  the  study  is  to  establish  a  good 
body  of  baseline  data  about  engineering  majors  who  have 
graduated  from  the  Naval  Academy.  Data  are  collected  from  21 
graduating  classes,  between  the  years  of  1985  and  2005.  This 
is  the  first  time  that  information  (feedback)  has  been 
gathered  from  Naval  Academy  graduates  that  provides  insight 
into  how  they  view  their  undergraduate  engineering 
education.  With  this  in  mind,  the  survey  was  designed  to 
gather  a  broad  base  of  data  about  what  professions  or  jobs 
graduates  are  currently  doing  and  how  their  undergraduate 
education  has  factored  into  their  careers. 

The  data  received  are  rich  for  exploration  and 
meaningful  comparison.  A  portion  of  this  study  focuses  on 
comparing  graduates  who  are  still  in  an  active  duty  military 
status  to  graduates  who  have  transitioned  to  the  civilian 
work  force,  with  regard  to  how  they  view  their  engineering 
education.  Comparisons  are  also  made  between  graduates  in 
technical  and  non-technical  careers,  as  well  as  across  the 
eight  different  engineering  majors  available  to  Naval 
Academy  students. 

C.  RESEARCH  QUESTIONS  AND  METHODOLOGY 

The  primary  research  guestion  is:  Utilizing  ABET's 
accreditation  criteria  for  the  Naval  Academy' s  engineering 
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program  as  a  means  of  measurement,  to  what  extent  do 
graduates  believe  that  their  undergraduate  education  in 
engineering  is  applicable  to  their  professional  lives,  and 
how  do  they  judge  their  level  of  preparation?  Some  secondary 
research  areas  entail  comparisons  between  civilian  and 
military  respondents,  as  well  as  between  technical  and  non¬ 
technical  jobs. 

This  is  a  quantitative  study.  As  one  of  the  purposes  of 
the  study  is  to  develop  broad  -  spectrum  baseline  data,  some 
of  the  analysis  is  devoted  to  descriptive  statistics  that 
illustrate  interesting  trends  or  highlight  areas  that  may  be 
rich  for  meaningful  comparisons.  Most  of  the  analysis  uses 
comparisons  between  group  means,  conducted  utilizing  paired 
sample  and  independent  sample  t-tests  and  analysis  of 
variance  tests  (ANOVA) .  These  tests  focus  on  respondents' 
answers  to  questions  concerning  the  applicability  of  and 
preparation  achieved  through  their  undergraduate  engineering 
education.  Again,  the  civilian  and  military  groups  are 
compared;  and  contrasts  are  made  across  the  engineering 
fields . 

D.  BENEFITS  OF  THE  STUDY 

There  are  several  possible  benefits  to  this  study.  The 

primary  benefit  is  the  establishment  of  baseline  data  about 

engineering  graduates  of  the  Naval  Academy  over  the  past  20 

years,  which  can  be  useful  in  several  ways.  It  gives  leaders 

at  the  Naval  Academy  information  they  can  in  turn  provide  to 

the  Accreditation  Board  for  Engineering  and  Technology,  as 

one  of  the  requirements  for  accreditation  renewal.  More 

importantly,  it  provides  information  that  can  be  used  as  a 

tool  for  engineering  program  improvement  at  the  Naval 
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Academy,  by  identifying  program  strengths  and  weaknesses. 
This  study  enables  leaders  at  the  Naval  Academy  to  assess 
how  well  we  are  preparing  graduates,  from  an  educational 
perspective,  for  their  professional  lives  in  the  Navy, 
Marine  Corps,  and  beyond. 

E.  ORGANIZATION  OF  THE  STUDY 

The  study  is  organized  into  five  chapters.  Chapter  I 
is  an  introduction  and  a  brief  explanation  of  the  study. 
Chapter  II  provides  a  literature  review  of  information 
relevant  to  the  history  and  background  of  ABET  and  the 
development  of  their  'learning  outcomes, '  an  overview  of 
approaches  to  program  evaluation,  the  accreditation 
practices  of  other  universities,  and  survey  implementation 
practices.  Chapter  III  provides  and  in  depth  look  at  the 
research  methodology  used,  along  with  a  description  of  the 
site  of  the  study  and  the  survey  instrument  that  is 
utilized.  Some  descriptive  statistics  characterizing  the 
sample  are  also  presented  in  chapter  three.  Chapter  IV  is 
dedicated  to  data  analysis.  Chapter  V  provides  the  results 
of  the  study  along  with  some  discussion  of  the  implications 
associated  with  the  results.  Chapter  V  also  discusses  some 
of  the  limitations  of  the  study,  and  ends  with 
recommendations  for  further  research.  Finally,  the 
appendices  include  the  survey  instrument  in  its  entirety, 
along  with  frequencies  and  descriptive  statistics  for  the 
data  set  that  will  be  useful  for  further  research. 
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II.  LITERATURE  REVIEW 


A.  INTRODUCTION 

This  chapter  begins  with  a  detailed  history  of  the 
Accreditation  Board  for  Engineering  and  Technology  (ABET) 
and  describes  the  recent  development  of  ABET's  current 
accreditation  standards,  known  as  Engineering  Criteria  2000 
(EC2000)  .  These  criteria  were  developed  in  response  to  both 
the  evolving  demands  of  the  engineering  profession  and 
advancements  in  the  realm  of  evaluation.  The  discussion  then 
turns  to  the  specific  aspect  of  EC2000  that  is  addressed  in 
this  study,  namely  Criterion  3,  or  Program  Outcomes 
Assessment.  After  defining  the  Criterion  3  outcomes,  this 
chapter  addresses  ways  in  which  to  choose  outcome 
indicators . 

The  chapter  then  addresses  the  concept  of  program 
evaluation,  with  specific  emphasis  on  survey  implementation 
and  utilization  as  a  tool  to  conduct  a  program  evaluation. 
Additional  methods  of  evaluation  are  briefly  discussed,  with 
emphasis  on  the  evaluation  of  learning  outcomes.  The  chapter 
continues  with  a  discussion  about  accreditation  practices  at 
various  universities  and  colleges,  as  well  as  design 
techniques  for  courses  that  satisfy  the  ABET  accreditation 
criteria.  The  chapter  concludes  with  a  brief  discussion  of 
teaching  ethics  as  part  of  engineering  curricula. 

B.  ABET  HISTORY 

The  Accreditation  Board  for  Engineering  and  Technology 
(ABET)  Web  site  provides  historical  background  for  the 
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organization  (ABET,  2007) .  ABET  (originally  named  the 
Engineers  Council  for  Professional  Development,  or  ECPD)  was 
founded  in  New  York  in  1932,  as  a  composite  of  seven 
engineering  societies  that  covered  all  the  major  engineering 
fields.  From  the  beginning,  the  organization  focused  on  the 
guidance,  training,  education  and  recognition  of  both 
engineering  students  and  academic  institutions.  The  ECPD 
developed  into  an  accreditation  society  almost  immediately 
following  its  inception,  and  in  1980  adopted  the  official 
name  ABET  in  order  to  reflect  this  emphasis  on 
accreditation . 

ABET  has  grown  to  include  28  professional  societies  and 
accredits  more  than  2700  programs  at  more  than  550  colleges 
and  universities,  including  seven  programs  at  the  United 
States  Naval  Academy.  These  include  Aerospace,  Electrical, 
Mechanical  and  Systems  Engineering,  accredited  since  1970, 
Naval  Architecture  and  Ocean  Engineering,  accredited  in 
1972,  and  Computer  Science  in  1987  (ABET,  2007) . 

Following  a  lengthy  development  period,  ABET  adopted  a 
revolutionary  approach  to  accreditation  with  its 
implementation  of  Engineering  Criteria  2000  (EC2000). 
Enacted  in  1997,  EC2000  has  allowed  ABET  to  shift  its  focus 
during  accreditation  procedures  to  "what  is  learned  rather 
than  what  is  taught"  (ABET,  2007) . 

C.  DEVELOPMENT  OF  EC2000 

By  the  end  of  the  20th  century,  the  curricula  of 
engineering  education  had  evolved  from  a  focus  on 
engineering  practice  and  application  toward  a  focus  on 
mathematics,  engineering  theory  and  applied  science  (Prados, 
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Peterson  &  Lattuca,  2005)  .  As  a  result,  a  gap  began  to 
develop  between  the  skill  demands  of  practicing  engineers  in 
the  industrial  workforce  and  the  skills  that  engineering 
education  was  providing  to  students.  Prados,  et  al  (2005) 
explain  that,  with  continuing  trends  toward  globalization 
and  advancements  in  the  information  technology  area,  many 
employers  began  to  recognize  that  sound  technical  skills 
alone  would  not  dictate  success  within  the  profession.  In 
particular,  employers  reported  the  need  for  young  engineers 
to  gain  proficiency  in  communication  skills,  quality 
assurance,  leadership  and  participation  in  teams  and  work 
groups,  commitment  to  continuous  education  and  learning, 
innovative  thinking  and  creativity. 

Until  the  development  of  EC2000,  ABET's  accreditation 
criteria  focused  on  quantitative  measurement  of  engineering 
program  inputs,  such  as  number  of  faculty  members  or  seat 
time  (number  of  hours  per  student)  in  a  particular  subject. 
These  criteria,  while  relevant  and  easy  to  measure  and  apply 
universally,  did  not  address  some  of  the  engineering  skills 
and  many  of  the  general  management  skills  associated  with 
success  in  the  profession.  ABET  evaluators  spent  their  time 
number  crunching  and  auditing  programs,  rather  than 
providing  qualitative  professional  assessment.  Not  until  the 
early  1990's  did  ABET  realize  that  their  strict  adherence  to 
measurement  of  program  inputs  had  made  them  partially 
responsible  for  the  widening  rift  between  the  needs  of  the 
engineering  practice  and  the  outcomes  of  engineering 
educational  programs.  After  a  substantial  review  process, 
ABET  released  the  new  EC2000  (Engineering  Criteria  2000) , 
which  shifted  emphasis  away  from  standardized  compliance  to 
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program  input  requirements  and  toward  and  emphasis  on 
defining  program  objectives  and  learning  outcomes  (Prados, 
Peterson  &  Lattuca,  2005)  . 

EC2000  was  piloted  at  five  different  institutions  in 
1996  and  1997.  Following  the  pilot  run,  EC2000  became 
optional  for  the  three-year  period  from  1998  -  2000.  During 
this  time,  institutions  could  volunteer  to  be  evaluated 
using  the  new  criteria,  or  continue  to  use  the  traditional 
ABET  criteria.  The  number  of  institutions  that  chose  to 
utilize  the  EC2000  criteria  increased  from  21%  in  1998  to 
83%  in  2000 (Prados,  Peterson  &  Lattuca,  2005)  . 

D .  PROGRAM  OUTCOMES 

EC2000  divides  the  criteria  for  program  accreditation 
into  eight  different  areas  that  are  listed  in  order  below, 
as  found  on  ABET's  official  Website  (ABET  Commission,  2006, 
pp .  1 - 3 )  : 

Criterion  1 :  Students 

Criterion  2:  Program  Educational  Objectives 

Criterion  3:  Program  Outcomes  Assessment 

Criterion  4:  Professional  Component 

Criterion  5:  Faculty 

Criterion  6:  Facilities 

Criterion  7 :  Institutional  Support  and  Financial 

Resources 

Criterion  8 :  Program  Criteria 

Feedback  from  graduates  is  a  vital  component  of  an 
engineering  program  review  when  attempting  to  satisfy 
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Criterion  3,  "Program  Outcomes  Assessment."  According  to 
Criterion  3,  ABET  requires  that  an  institution  be  able  to 
produce,  assess  and  document  the  achievement  of  designated 
"skills,  knowledge  and  behaviors"  (ABET,  2005,  p.  1)  that  a 
student  should  achieve  by  the  time  of  graduation.  These 
skills,  knowledge  and  behaviors  are  further  categorized  as 
items  'a'  through  'k, '  listed  below  in  Table  1: 


Table  1 :  Criterion  3  Program  Outcomes 


a 

an  ability  to  apply  knowledge  of  mathematics,  science  and 

engineering 

b 

an  ability  to  design  and  conduct  experiments,  as  well  as  to 
analyze  and  interpret  data 

c 

an  ability  to  design  a  system,  component,  or  process  to  meet 
desired  needs  within  realistic  constraints  such  as  economic, 
environmental,  social,  political,  ethical,  health  and  safety, 
manufacturability,  and  sustainability 

d 

an  ability  to  function  on  multi-disciplinary  teams 

e 

an  ability  to  identify,  formulate  and  solve  engineering  problems 

f 

an  understanding  of  professional  and  ethical  responsibility 

g 

an  ability  to  communicate  effectively 

h 

the  broad  education  necessary  to  understand  the  impact  of 
engineering  solutions  in  a  global,  economic,  environmental  and 
societal  context 

i 

recognition  of  the  need  for,  and  ability  to  engage  in  life-long 
learning 

j 

a  knowledge  of  contemporary  issues 

k 

an  ability  to  use  the  techniques,  skills,  and  modern  engineering 
tools  necessary  for  engineering  practice. 

An  important  aspect  of  ABET's  EC2000  is  that  it  shifts 

focus  from  an  emphasis  on  program  inputs  to  a  focus  on 

program  outcomes.  For  our  purposes,  it  may  be  useful  to 

define  program  inputs,  processes,  outputs  and  outcomes,  and 
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emphasize  the  difference  between  outputs  and  outcomes. 
Rogers  (2000)  provides  useful  definitions  of  these  terms. 
Inputs  are  those  things  that  are  brought  into  an  engineering 
program  by  students  (test  scores,  credentials, 
competencies) ,  faculty  (credentials,  experience,  values)  and 
the  institution  itself  (facilities,  equipment,  resources) . 
Processes  focus  on  what  the  institution  does  to  exert 
influence  on  and  work  with  inputs.  These  include  student 
choice  of  major  and  courses,  faculty  teaching  load  and  class 
size,  and  institutional  procedures  and  governance.  Outputs 
can  be  seen  as  the  quantitative,  concrete  results  of  the 
processing  of  inputs.  These  include  grades,  retention  and 
employment  statistics.  Outcomes,  on  the  other  hand,  are  less 
tangible.  They  are  the  knowledge,  skills,  attitudes,  values 
and  behaviors  that  a  student  develops  as  the  result  of 
having  gone  through  the  educational  program  (Rogers,  2000)  . 

Rogers  (2000)  also  illustrates  the  increased  level  of 
difficulty  when  an  institution  moves  from  classroom 
assessment  to  program  assessment.  The  problems  lie  with  the 
degree  of  complexity,  time  span,  level  of  specificity, 
accountability,  level  of  faculty  buy-in  and  cost  (Rogers, 
2000)  .  Evaluating  an  entire  engineering  program  is  more 
complex  than  evaluating  a  course,  because  the  desired 
outcomes  are  combined  across  the  entire  spectrum  of  the 
curriculum.  It  takes  much  more  time  to  evaluate  a  program, 
because  a  student  must  be  allowed  to  complete  the  program  in 
its  entirety.  Program  assessment  is  less  specific  in  nature, 
and  it  is  hard  to  assign  accountability  for  success  or 
failure  to  individuals  within  the  program.  Finally,  program 
assessment  is  much  more  costly  in  terms  of  time,  money  and 
resources . 
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E. 


CHOOSING  OUTCOME  INDICATORS 


Periodically,  the  U.S.  Naval  Academy's  Engineering 
department  must  "renew"  the  accreditation  of  its  entire 
program.  The  process  is  run  by  the  Accreditation  Board  of 
Engineering  and  Technology  (ABET) ;  it  requires  a  thorough 

review  of  the  program  from  many  different  aspects,  one  of 
which  is  an  assessment  of  ABET' s  "student  learning 
outcomes."  While  there  are  several  ways  to  evaluate  these 

outcomes,  the  evaluation  method  most  relevant  to  this  study 
is  feedback  from  graduates.  In  their  responses  to  a  survey 
distributed  in  the  fall  of  2005,  graduates  had  the 
opportunity  to  assess  the  value  of  their  undergraduate 

education . 

Prior  to  2005,  there  had  never  been  an  external 

evaluation  of  the  U.S.  Naval  Academy's  Engineering  program 
analyzing  feedback  from  graduates.  With  ABET' s  new  "student 
learning  outcomes"  as  a  basis  for  this  study,  the  intention 
is  to  determine  to  what  extent  graduates  believe  these 
outcomes  are  or  are  not  applicable  to  their  professional 
lives  and  to  what  degree  these  learning  outcomes  were 
successfully  achieved  during  their  four  years  at  the 
Academy . 

ABET  is  the  recognized  accreditation  source  for  all  of 
the  Engineering  programs  in  the  country  (Brizendine,  2004)  . 
It  is  a  federation  of  32  professional  and  technical 
societies  that  has  been  functioning  for  more  than  70  years. 
Every  six  years,  an  Engineering  program  that  desires 
accreditation  is  required  to  satisfy  ABET's  criteria  and 
submit  the  results  to  an  ABET  Board  of  Visitors  for 
approval.  Brizendine  (2004)  tells  us  that,  historically,  the 
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accreditation  has  been  based  primarily  on  the  capabilities 
and  the  capacities  of  the  program  itself.  In  1996,  however, 
ABET  introduced  an  entirely  new  method  (in  addition  to, 
rather  than  in  place  of  those  already  in  use)  to  evaluate 
the  quality  of  a  given  Engineering  program.  This  new  method 
requires  an  evaluation  of  program  outcomes  (Felder,  Brent, 
2003)  . 

ABET's  guidelines  for  measuring  outcomes  are  based  on 
their  list  of  eleven  "student  learning  outcomes"  (identified 
as  'a'  through  'k,  '  see  Table  1),  which  were  implemented  as 
part  of  their  Technology  Criteria  2000  (Brizendine,  2004). 
These  outcomes  are  subject  to  a  good  deal  of  interpretation 
by  the  Engineering  departments  of  the  various  institutions, 
but  McGourty,  Besterf ield-Sacre  and  Shuman  (1998)  attempt  to 
provide  a  way  to  categorize  them.  They  divide  these  desired 
outcomes  into  three  types:  cognitive  (what  we  "know"), 
behavioral  (what  we  "do") ,  and  attitudinal  (what  we  "think" 
or  "feel") .  The  authors  believe  these  outcomes  to  be 
measurable,  but  they  reason  that  there  is  no  score  that  can 
be  applied  universally  to  determine  passing  or  failing. 
McGourty,  et  al .  (1998)  believe  that  the  advantage  to  vague 
wording  of  the  "learning  outcomes"  is  that  it  leaves  it  up 
to  the  individual  Engineering  departments  to  best  define 
them  (and  evaluate  them)  in  order  to  align  the  outcomes  with 
their  unique  program  needs. 

Aft  (2002)  explains  that  there  are  several  different 
ways  to  evaluate  the  "student  learning  outcomes."  An 
Engineering  program  can  conduct  an  internal  self-study, 
which  relies  on  inputs  and  feedback  from  faculty  and  current 
students  to  assess  the  program.  Equally  important  is  the 
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concept  of  external  assessment  methods,  which  could  include 
the  periodic  ABET  visit  or  the  survey  of  graduates  to 
evaluate  the  program.  Aft  (2002)  reasons  that  a  quality 
program  will  utilize  a  variety  of  methods  for  both  internal 
and  external  assessment. 

There  has  been  research  conducted  on  how  to  best 
implement  and  evaluate  the  "student  learning  outcomes"  on  an 
internal  level.  Specifically,  the  goal  has  been  to  take 
feedback  from  faculty,  staff  and  students  enrolled  in  the 
program  and  utilize  that  information  to  effect  positive 
change.  Felder  and  Brent  (2003)  tell  us  that  this  results  in 
"change  from  within"  in  terms  of  clarified  learning 
objectives,  and  improvements  in  instruction  and  assessment. 
There  have  also  been  studies  conducted  on  successful  methods 
for  externally  evaluating  an  Engineering  program,  as  will  be 
discussed  later  in  this  chapter. 

Scales,  Owen,  Shiohare  and  Leonard  (1998)  assert  that 
evaluation  begins  when  the  faculty  and  staff  of  a  program 
identify  the  desired  outcomes  of  a  course  or  program,  and 
then  identify  which  indicators  will  be  used  to  measure  these 
outcomes.  Indicators  can  be  classified  in  different  ways, 
depending  on  what  is  to  be  measured  (i.e.,  attitudes, 
behaviors,  knowledge,  performance)  and  how  it  is  to  be 
measured  (qualitative  vs.  quantitative),  or  even  by  who  is 
doing  the  measuring  (i.e.,  exit  surveys  given  by  the 
institution,  universal  FE  exams  given  nationally)  (Scales, 
Owen,  Shiohare  &  Leonard,  1998)  .  Availability  of  time  and 
financial  resources  may  be  the  biggest  determinant  when 
selecting  an  indicator. 
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Scales  et  al .  (1998)  identify  seven  different  types  of 

outcome  indicators,  one  of  which  is  alumni  surveys.  Alumni 
surveys  are  advantageous  because  they  are  relatively 
inexpensive  to  administer,  results  can  be  analyzed  quickly 
(if  quantitative  data  are  collected)  ,  and  they  allow  an 
institution  a  way  to  maintain  contact  with  its  graduates. 
Alumni  can  provide  useful  biographical  data  and  feedback 
concerning  the  quality  and  content  of  their  education,  and 
an  institution  can  track  how  these  evaluations  change  over 
time  (Scales  et  al . ,  1998). 

Unfortunately,  there  are  also  several  drawbacks  when 
using  alumni  surveys.  They  suffer  from  low  response  rates 
and  are  not  ideal  for  addressing  complex  issues,  as  there  is 
no  means  to  answer  respondents''  questions  if  there  is  an 
unclear  or  misunderstood  survey  question.  Similarly,  there 
is  no  way  to  guard  against  untruthful  or  misleading 
responses.  Thus,  alumni  surveys  are  most  effective  when 
designed  as  brief,  simple  questionnaires  (Scales  et  al . , 
1998) .  Overall,  Scales  et  al .  (1998)  report  that  in  a  survey 

given  to  the  representatives  of  engineering  programs 
attending  the  1997  Best  Assessment  Processes  in  Engineering 
Education:  A  Working  Symposium  conference,  the  results 

indicated  that  alumni  surveys  were  both  highly  utilized  and 
believed  to  be  very  useful. 

F.  EVALUATION 

Patton  (1987)  offers  an  excellent  distinction  of  the 

differences  between  formative  and  summative  evaluations. 

Formative  evaluations  are  conducted  in  order  to  allow  for 

program  improvement,  whereas  summative  evaluations  are 

utilized  to  determine  overall  program  effectiveness, 
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specifically  useful  when  making  a  determination  as  to 
whether  a  program  should  continue  or  not  (Patton,  1987)  . 
When  a  program  is  looking  to  improve  quality,  a  formative 
evaluation  should  be  utilized.  This  allows  a  program  to 
identify  strengths  (what  is  working  well)  and  weaknesses 
(what  processes  are  in  need  of  improvement) .  Also  central  to 
the  idea  of  formative  assessment  is  the  concept  of  feedback 
not  only  must  feedback  be  continuously  given  by  the 
evaluator,  but  that  feedback  must  also  be  acted  upon  in  such 
a  manner  as  to  improve  program  guality.  In  other  words,  the 
processes  of  evaluation  and  providing  feedback  themselves 
become  an  integral  part  of  continuous  improvement  of  program 
quality  (Patton,  1987). 

Patton  (1982)  further  clarifies  the  distinction  between 
formative  and  summative  evaluation  by  pointing  out  that 
summative  evaluation  results  in  a  judgment  being  made  about 
the  composite  worth  or  effectiveness  of  a  program.  Summative 
evaluations,  in  general,  tend  to  focus  on  program  outcomes, 
and  formative  evaluations  tend  to  focus  on  program 
processes.  Patton  is  quick  to  point  out  that  there  may  be 
some  blurring  of  the  lines  when  deciding  exactly  what  to 
focus  on  (process  or  outcome?)  ,  and  that  the  important 
distinction  occurs  when  deciding  on  the  purpose  of  the 
evaluation  (Patton,  1982). 

Royse,  Thyer,  Padgett  and  Logan  (2001)  discuss  the  fact 
that,  although  many  experts  believe  formative  evaluation  to 
be  synonymous  with  process  evaluation,  this  is  not 
necessarily  true.  The  key  difference  is  that  a  formative 
evaluation  is  typically  conducted  early  in  the  development 
of  a  program,  whereas  a  process  evaluation  can  be  conducted 
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at  any  time,  even  when  the  program  has  ended  (Royse  et  al . , 
2001) .  Again,  it  is  important  to  note  that  the  intention  of 
the  evaluation  is  still  program  improvement,  and  to 
differentiate  between  process  and  formative  evaluation  is 
simply  a  question  of  when  the  evaluation  occurs  within  the 
lifetime  of  the  program. 

Patton  (1987)  describes  the  inherent  tradeoffs  that  are 
inescapably  present  when  conducting  any  type  of  evaluation. 
Tradeoffs  are  unavoidable,  due  to  realistic  challenges 
associated  with  working  with  constrained  resources  and  time. 
One  particularly  relevant  tradeoff  scenario  that  Patton 
addresses  is  the  question  of  depth  versus  breadth.  Within 
the  confines  of  constrained  resources  and  time,  an  evaluator 
must  at  some  point  make  decisions  about  how  broad  or  how 
deep  the  evaluation  will  run.  Exploratory  studies,  Patton 
says,  lend  themselves  more  toward  a  focus  on  breadth,  which 
allows  for  a  wide  range  of  baseline  data;  a  focus  on  depth 
and  detail  may  be  more  desirable  when  evaluating  specific 
experiences  or  outcomes  (Patton,  1987). 

Another  key  consideration  in  evaluation  is  units  of 
analysis  to  be  studied.  On  the  topic,  Patton  (1987)  states 
that  "the  key  factor  in  selecting  and  making  decisions  about 
the  appropriate  unit  of  analysis  is  to  decide  what  unit  it 
is  that  you  want  to  be  able  to  say  something  about  at  the 
end  of  the  evaluation"  (p.  51)  .  Thus,  making  choices  about 
which  units  of  analysis  will  be  used  during  an  evaluation 
can  also  help  to  answer  questions  concerning  breadth  versus 
depth . 

The  process  of  obtaining  a  target  sample  is  of  great 
importance  in  evaluation.  Of  many  different  sampling 
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methods,  Patton  (1987)  provides  some  insight  into  one  that 
is  of  particular  importance  to  this  study  -  criterion 
sampling.  Criterion  sampling  occurs  when  all  chosen  cases 
"meet  some  predetermined  criterion  of  importance"  (Patton, 
1987,  p.  56)  .  For  example,  identifying  all  graduates  of  a 
university  who  majored  in  an  engineering  field  would 
constitute  criterion  sampling.  The  purpose,  Patton  (1987) 
tells  us,  is  "to  be  sure  to  understand  cases  which  are 
likely  to  be  rich  in  information, "  because  their  information 
will  likely  lead  evaluators  toward  program  improvement  (p. 
56)  . 

Patton  (1982)  also  highlights  the  importance  of 
determining  specifically  what  information  is  required  prior 
to  implementing  a  survey  or  questionnaire.  He  discusses  four 
different  types  of  questions  that  can  be  asked,  based  on 
what  an  evaluator  wishes  to  find  out:  (a) behavior  questions 
inquire  about  actions  (what  people  do)  ;  (b)  opinion 

questions  inquire  about  thoughts  (what  people  think) ;  (c) 

feeling  questions  inquire  about  emotions  (what  people  feel) ; 
and  (d)  knowledge  questions  inquire  about  facts  (what  people 
know)  (1982)  .  For  purposes  of  this  study,  the  focus  is 
primarily  on  the  thoughts  and  opinions  of  survey 
respondents . 

G.  BEST  PRACTICES  FOR  SURVEY  IMPLEMENTATION 

Couper,  Traugott  and  Lamias  (2001)  conducted  a  study  at 

the  University  of  Michigan  to  identify  some  traits  of  Web  - 

based  surveys  that  can  either  add  to  or  detract  from  their 

effectiveness  with  respect  to  response  rates  and  data 

quality.  The  authors  recognize  the  growing  trend  of  Web  - 

based  survey  utilization,  and  they  identify  some  key 
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advantages,  such  as  a  decrease  in  turnaround  time  between 
questionnaire  delivery  and  response  and  ease  of  access  to  a 
large  population  through  the  Web.  The  primary  focus, 
however,  is  on  the  pros  and  cons  of  various  types  of  Web 
survey  design. 

In  one  experiment,  the  effect  of  including  a  progress 
indicator  for  the  survey  respondent  was  studied.  While  a 
larger  percentage  of  respondents  completed  the  survey  if 
they  were  able  to  track  their  progress  toward  completion, 
the  progress  indicator  also  involved  a  more  complex  survey 
design  which  led  to  an  increase  in  survey  completion  time 
(Couper  et  al . ,  2001) .  In  another  experiment,  the  effects  of 
placing  single  versus  multiple  items  on  a  page  were  studied. 
When  a  single  item  was  placed  on  each  page,  the  survey  took 
longer  to  complete;  when  multiple  items  were  placed  on  each 
page,  response  time  decreased,  but  the  correlation  between 
answers  increased  (Couper  et  al . ,  2001)  .  The  third 
experiment  compared  the  response  rates  when  respondents  were 
given  the  option  of  either  entering  data  in  a  text  field  or 
using  the  computer's  mouse  to  click  on  a  button.  Response 
rates  were  significantly  higher  when  respondents  could  use 
the  mouse  to  enter  all  information,  rather  than  entering 
data  into  a  text  field  (Couper  et  al . ,  2001) . 

These  types  of  experiments  seem  to  indicate  that,  when 
faced  with  the  task  of  designing  and  implementing  a  Web  - 
based  survey,  the  designer  will  have  to  make  choices  based 
on  what  type  of  information  they  are  attempting  to  retrieve. 
There  will  be  trade-offs  associated  with  each  choice. 

Schwarz  (1995)  tells  us  that  "respondents  use  features 
of  the  questionnaire  to  determine  the  meaning  of  a  question 


18 


and  to  generate  a  useful  answer"  (p.  161)  .  Regardless  of 
whether  it  is  intended  to  be  unbiased  or  neutral,  there  are 
certain  aspects  of  a  survey  that  force  respondents  to  make 
interpretations  and  judgments.  For  example,  the  ordering  of 
questions  can  have  an  impact  or,  in  the  case  of  a  survey 
that  uses  a  numerical  rating  scale,  the  words  used  to  define 
the  extreme  high  and  low  values  on  the  scale  can  influence 
respondents,  as  can  the  numerical  values  chosen  to  represent 
different  ratings  (Schwarz,  1995) .  For  example,  in  one 
experiment  conducted  by  Schwarz  (1995),  respondents  to  a 
survey  were  statistically  much  less  likely  to  report  a  value 
below  the  midpoint  on  a  rating  scale  if  the  scale  ranged 
from  -5  to  5  as  opposed  to  a  scale  that  ranged  from  0  to  10. 

H.  ACCREDITATION  AT  OTHER  INSTITUTIONS 

McGourty,  Sebastian  and  Swart  (1998)  describe  the 
process  that  one  institution,  the  New  Jersey  Institute  of 
Technology  (NJIT) ,  implemented  in  order  to  achieve  a 
comprehensive  internal  assessment  of  engineering  education. 
The  team  at  NJIT  followed  a  five-step  process  to  develop 
their  program.  The  first  step,  defining  objectives, 
strategies,  and  outcomes,  required  the  team  to  identify  the 
students  and  courses  to  which  the  assessment  program  would 
apply;  then,  faculty  input  (through  focus  groups  and 
surveys)  was  used  to  decide  which  student  learning  outcomes 
would  be  assessed. 

The  outcomes  chosen  align  with  the  outcomes  associated 
with  ABET's  EC-2000.  The  second  step  required  the  selection 
of  assessment  methods,  with  a  focus  on  reliability. 


validity,  fairness  and,  perhaps  most  importantly,  choosing 

only  those  methods  that  could  provide  information  useful  to 
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continuous  program  improvement.  Step  three  was  a  pilot  run 
of  the  assessment  process;  step  four  was  full  implementation 
of  the  assessment  process.  The  goal  of  step  five  was  to 
close  the  loop  on  the  process  by  applying  the  feedback  from 
the  assessment  toward  improvement  of  both  the  academic 
curriculum  and  the  assessment  process  itself  (McGourty, 
Sebastian  &  Swart,  1998). 

There  were  four  assessment  methods  chosen  by  the  NJIT 
team  at  the  start  of  this  program.  The  process  began  with  a 
survey  given  to  students  prior  to  beginning  a  course.  The 
students  would  then,  with  help  from  faculty,  maintain  a 
portfolio  that  tracked  their  progress  through  the  courses 
chosen  for  assessment.  A  peer  assessment  tool  (the  "Student 
Developer")  allowed  students  to  give  feedback  to  each  other 
throughout  the  curriculum.  Finally,  at  the  end  of  the 
course,  the  faculty  would  observe  and  evaluate  an  oral 
presentation  given  by  individual  students,  which  would  allow 
faculty  to  assess  student  outcomes  for  the  course.  Data 
useful  in  both  assessing  student  learning  and  improving 
program  composition  were  obtained  from  all  methods  with  the 
exception  of  the  portfolio,  which  was  deemed  to  be  too 
laborious  for  students  and  faculty  and  subsequently  dropped 
(McGourty,  Sebastian  &  Swart,  1998). 

Regan  and  Schmidt  (1999)  outline  the  evaluation  process 
used  by  the  University  of  Maryland,  College  Park,  to 
determine  if  the  "student  learning  outcomes"  are  achieved. 
Their  evaluation  method,  which  spanned  the  time  period  from 
entry  into  the  program  until  five  years  after  graduation, 
culminated  with  a  survey  of  graduates  to  determine  if  the 
"student  learning  outcomes"  were  applicable  in  their 
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professional  lives  and  to  what  extent  those  outcomes  had 
been  achieved  during  their  undergraduate  education.  Their 
results  provided  them  with  a  source  of  external  feedback 
that  proved  to  be  beneficial  when  attempting  to  evaluate  the 
achievement  of  ABET's  student  learning  outcomes. 

I.  OTHER  METHODS  OF  EVALUATING  LEARNING  OUTCOMES 

In  this  section,  different  methods  of  assessing 
learning  outcomes  are  discussed.  The  majority  of  the 
assessment  methods  were  developed  specifically  with  ABET's 
EC2000  criteria  in  mind,  for  use  in  determining  the  level  to 
which  Criterion  Three  (See  Table  1)  was  attained. 

Christy  and  Lima  (1998)  advocate  a  method  of  assessing 
student  learning  that  could  possibly  be  useful  as 
institutions  attempt  to  measure  EC2000  Criterion  Three,  or 
student  learning  outcomes.  This  method  involves  the  use  of 
portfolios  maintained  by  the  individual  students  throughout 
their  time  in  the  engineering  program.  One  section  of  the 
portfolio  would  list  course-specific  desired  goals  or 
outcomes  for  students.  Periodically,  the  students  would 
evaluate  their  progress  toward  mastery  of  the  outcomes, 
based  on  self  -  evaluation  of  their  work  in  the  course. 

The  intention  is  to  allow  the  student  to  take 
responsibility  for  their  own  education;  with  the  help  of  the 
portfolio  process  of  documentation  and  criterion  -  based 
self  assessment  to  guide  them  toward  the  achievement  of  the 
desired  outcomes.  In  a  study  conducted  at  Ohio  State 
University,  both  freshmen  and  senior  engineering  students 
were  required  to  maintain  portfolios  in  one  particular 
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class.  78%  of  the  seniors  and  80%  of  the  freshmen  reported 
that  the  portfolios  did  in  fact  enhance  their  learning 
experience  (Christy  &  Lima,  1998) . 

One  relatively  unexplored  method  of  evaluating  the  more 
technical  of  ABET's  learning  outcomes  is  the  nationally 
standardized  Fundamentals  of  Engineering  (FE)  Exam  (Watson, 
1998)  .  This  eight  -  hour  test  covers  15  different  areas  of 
technical  engineering  science,  and  is  administered  to 
graduating  engineering  students  who  may  wish  to  continue 
toward  a  professional  engineering  career.  While  this  test 
could  provide  some  rich  data  with  which  to  assess  student 
learning,  it  is  rendered  ineffective  as  a  measurement  tool 
by  several  factors.  First,  the  National  Council  of  Examiners 
for  Engineers  and  Surveyors  (NCEES)  is  bound  by  privacy  and 
release  restrictions.  Second,  even  if  NCEES  were  allowed  to 
furnish  test  results,  the  test  is  taken  on  a  predominantly 
voluntary  basis.  Many  engineering  students  never  take  the 
exam,  and  those  who  do  take  it  tend  to  be  highly  motivated 
(Watson,  1998)  . 

J.  DESIGN  TECHNIQUES  FOR  COURSES  THAT  SATISFY  ABET 

Shaeiwitz  (1998)  discusses  the  important  role  that 
classroom  assessment  can  play  in  the  development  of  a 
comprehensive  program  evaluation.  In  particular,  the  author 
discusses  the  importance  of  nested  feedback  loops,  in  which 
measurement  of  outcomes  and  feedback  occur  regularly 
throughout  the  students'  progression  through  a  curriculum. 
Using  the  concept  of  nested  feedback  loops,  engineering 
programs  can  benefit  from  both  formative  and  summative 
assessment . 
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In  formative  assessment,  the  audience  is  a  particular 
instructor  or  faculty  member,  and  the  goal  is  to  improve 
student  learning  at  the  course  level  (Shaeiwitz,  1998)  . 
Formative  assessments  represent  short-term  feedback  loops, 
where  the  time  passage  between  measurement  and  feedback  is 
minimized. 

With  summative  assessment,  the  goal  is  to  evaluate  the 
effectiveness  of  the  educational  curriculum  taken  as  a 
whole,  and  the  audience  is  external  to  the  department. 
Summative  assessment  evaluates  the  entire  educational 
process,  from  beginning  to  end.  Shaeiwitz  (1998)  believes 
that  a  program  designed  to  satisfy  ABET's  EC2000  criteria 
will  take  advantage  of  nested  feedback  loops,  which  provide 
multiple  opportunities  for  formative  assessment  and  program 
improvement . 

K.  ETHICS  IN  ENGINEERING  EDUCATION 

Haws  (2001)  discusses  the  importance  of  teaching  ethics 
to  engineers,  as  a  way  to  help  students  to  think 
divergently.  By  this,  he  implies  that  it  is  important  to 
gain  a  broader  perspective  by  looking,  through  the  eyes  of  a 
non-engineer,  at  the  potential  impacts  of  decisions  on  a 
wide  range  of  people  and  things.  Teaching  the  Professional 
Code  of  Ethics  solely  by  presenting  the  ethical 
considerations  summarized  in  that  document  is  insufficient 
(Haws,  2001) .  A  good  understanding  of  ethical  considerations 
will  most  likely  be  achieved  through  a  number  of  different 
approaches,  including  professional  codes,  humanist  readings, 
theoretical  grounding,  ethical  heuristics,  case  studies  and 
service  learning  (actively  applying  ethics  to  community- 
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based  projects) .  Haws  (2001)  recognizes  value  in  all  six  of 
these  methods,  but  favors  a  combination  of  theoretical 
grounding  and  service  learning. 

L .  SUMMARY 

This  chapter  has  covered  the  history  of  the 
Accreditation  Board  for  Engineering  and  Technology  (ABET) , 
highlighting  the  relatively  recent  changes  that  ABET  has 
made  to  their  accreditation  criteria.  The  changes  most 
relevant  to  this  study  involve  the  evaluation  of  ABET's 
Criterion  Three  'Student  Learning  Outcomes.''  Best  practices 
for  choosing  and  evaluating  indicators  of  these  outcomes 
were  explored  in  detail.  This  chapter  also  explored  the  use 
of  various  methods  of  evaluation,  with  specific  emphasis  on 
surveys  as  tools  for  program  evaluation.  Finally,  some 
background  was  given  concerning  accreditation  at  other 
institutions . 
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Ill .  METHODOLOGY 


A.  INTRODUCTION 

There  are  several  purposes  to  this  study.  One  goal  is 
to  establish  baseline  data  about  engineering  graduates  of 
the  United  States  Naval  Academy  over  the  past  20  years.  The 
bulk  of  the  analysis  is  devoted  to  an  overall  assessment  of 
the  relevance  of  engineering  education  to  Naval  Academy 
graduates  and  the  level  of  preparedness  afforded  them 
through  their  undergraduate  education. 

Additionally,  an  attempt  is  made  to  explore  the 
differences  (if  any)  that  exist  between  military  and 
civilian  graduates  of  the  United  States  Naval  Academy  who 
majored  in  engineering,  with  respect  to  their  level  of 
preparedness  for  their  profession  and  the  level  of 
applicability  of  their  undergraduate  education  to  their 
professional  lives.  Finally,  comparisons  are  made  between 
graduates  in  technical  and  non-technical  career  fields,  as 
well  as  across  the  different  engineering  majors. 

The  Accreditation  Board  for  Engineering  and  Technology 
(ABET)  Criterion  Three  'Student  Learning  Outcomes'  (See 
Table  1)  provide  the  basis  for  all  analyses  and  comparisons, 
and  this  chapter  provides  background  information  on  how  the 
study  was  conducted.  The  site  of  the  study,  survey 
instrumentation  and  implementation  procedure,  sample 
description  and  analysis  procedures  are  all  covered  in 
detail . 
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B.  SITE  OF  STUDY 

This  study  was  conducted  at  the  United  States  Naval 
Academy  in  Annapolis,  MD.  The  Naval  Academy  is  a  four-year 
institution  whose  graduates  are  awarded  both  a  Bachelor  of 
Science  degree  and  an  officer's  commission  into  the  Navy  or 
Marine  Corps  as  either  ensigns  or  second  lieutenants.  The 
minimum  active  duty  service  requirement  following  graduation 
is  five  years;  many  graduates  remain  on  active  duty  past  the 
minimum  commitment,  and  many  take  their  education  and 
military  experience  and  join  the  civilian  workforce. 

Between  the  years  1990  -  2005,  an  average  of  1246  men 
and  women  were  admitted  to  the  Naval  Academy  each  year. 
During  this  same  time  span,  an  average  of  964  men  and  women 
graduated  and  were  commissioned  into  the  Navy  or  Marine 
Corps  each  year.  The  average  attrition  rate  for  each 
incoming  class  was  22.5%,  which  means  that  an  average  of 
77.5%  of  all  men  and  women  admitted  to  the  Naval  Academy 
went  on  to  graduate  during  this  time  period  (U.S.  Naval 
Academy  Institutional  Research,  2004). 

There  are  19  different  majors  offered  at  the  Naval 
Academy  in  areas  of  Engineering  and  Weapons,  Mathematics  and 
Science,  and  Humanities  and  Social  Sciences.  Selection  of 
major  is  voluntary,  and  an  average  of  35%  of  each  class  of 
Midshipmen  will  choose  to  major  in  an  engineering  field.  For 
this  study,  the  focus  is  on  graduates  who  studied  in  one  of 
eight  available  engineering  majors.  In  particular,  this 
study  deals  only  with  engineering  majors  who  graduated 
between  the  years  of  1985  -  2005.  During  this  20  -  year  time 
span,  a  total  of  7572  graduates  of  the  Naval  Academy  majored 
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in  some  field  of  engineering.  For  purposes  of  this  study, 
these  7572  graduates  are  the  population. 

C .  INSTRUMENTATION 

The  survey  instrument  is  essentially  broken  up  into 
three  distinct  sections.  The  intent  is  to  track  information 
about  the  professional  lives  of  respondents  between  the  time 
of  their  graduation  and  present,  then  gather  information 
about  the  applicability  of  their  undergraduate  education  to 
their  career  and  gauge  the  level  of  preparation  each 
respondent  feels  was  achieved  through  their  undergraduate 
education . 

The  survey  (See  Appendix  A  for  the  complete  version) 
begins  with  13  questions  designed  to  gather  information 
about  the  current  professional  career  status  of  each 
respondent.  If  respondents  indicate  that  they  are  in  an 
active  duty  status,  they  are  asked  to  give  information  such 
as  warfare  community,  rank,  technical  nature  of  current  job 
and  career  intentions.  If  respondents  indicate  that  they  are 
working  in  the  civilian  professional  workforce,  they  are 
asked  to  provide  information  such  as  current  title  and/or 
position,  name  of  employer,  technical  nature  of  current  job 
and  employing  organization,  and  number  of  years  served  in 
current  position. 

A  field  of  24  guestions  was  designed  for  the  second 
section  of  this  survey,  with  the  intention  of  addressing 
ABET's  accreditation  criteria.  Specifically,  these  guestions 
were  developed  in  order  to  ascertain:  (a)  The  level  of 
preparedness  that  each  graduate  believes  was  achieved  during 
their  undergraduate  engineering  education  with  respect  to 
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ABET' s  Criterion  3  'Program  Outcomes'’  ('a'  through  'k'  )  ; 
and  (b)  The  degree  to  which  each  of  the  'a'  through  'k' 
criteria  is  relevant  to  their  current  positions. 

According  to  Criterion  3,  ABET  requires  that  an 
institution  be  able  to  produce,  assess  and  document  the 
achievement  of  designated  "skills,  knowledge  and  behaviors" 
(ABET,  2005,  p.  1)  that  a  student  should  achieve  by  the  time 
of  graduation.  These  skills,  knowledge  and  behaviors  are 
further  categorized  as  items  "a"  through  "k":  (a)  an  ability 
to  apply  knowledge  of  mathematics,  science  and  engineering; 
(b)  an  ability  to  design  and  conduct  experiments,  as  well  as 
to  analyze  and  interpret  data;  (c)  an  ability  to  design  a 
system,  component,  or  process  to  meet  desired  needs  within 
realistic  constraints  such  as  economic,  environmental, 
social,  political,  ethical,  health  and  safety, 
manufacturability,  and  sustainability;  (d)  an  ability  to 
function  on  multi-disciplinary  teams;  (e)  an  ability  to 
identify,  formulate  and  solve  engineering  problems;  (f)  an 
understanding  of  professional  and  ethical  responsibility; 
(g)  an  ability  to  communicate  effectively;  (h)  the  broad 
education  necessary  to  understand  the  impact  of  engineering 
solutions  in  a  global,  economic,  environmental  and  societal 
context;  (i)  recognition  of  the  need  for,  and  ability  to 
engage  in  life-long  learning;  (j)  a  knowledge  of 
contemporary  issues;  and  (k)  an  ability  to  use  the 
techniques,  skills,  and  modern  engineering  tools  necessary 
for  engineering  practice. 

The  second  section  of  the  survey  includes  questions 
that  identify  the  above  skills  or  capabilities  and  asks 
respondents  to  indicate  both  the  level  of  applicability  of 
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those  skills  to  their  professional  jobs  (12  questions)  and 
the  level  of  preparation  in  each  of  those  skill  areas 
afforded  them  by  their  undergraduate  engineering  education 
(12  questions) .  Respondents  were  asked  to  indicate  the  level 
of  applicability  and  preparation  on  a  five-point  scale, 
ranging  from  "Not  at  all"  to  "Extremely"  applicable  or 
prepared.  The  table  below  illustrates  the  manner  in  which 
these  24  questions  can  be  mapped  to  ABET's  Criterion  3 
"Program  Outcomes." 

Table  2 :  Mapping  Survey  Questions  to  ABET  Criterion  3 

"Program  Outcomes" 


Survey  Question 

Addresses  which  ABET  Criterion  3 
"Program  Outcome" 

Ability  to  apply  knowledge  of  math, 
engineering  and  science? 

a.  an  ability  to  apply  knowledge 
of  mathematics,  science  and 
engineering 

Ability  to  analyze  and  interpret 
data? 

b.  ability  to  design  and  conduct 
experiments,  analyze  and 
interpret  data 

Ability  to  identify,  formulate  and 
solve  engineering  problems? 

e.  ability  to  identify, 
formulate  and  solve  engineering 
problems 

Ability  to  design  systems, 
components  or  processes  to  meet 
needs? 

c.  ability  to  design  a  system, 
component,  or  process  to  meet 
desired  needs  within  realistic 
constraints 

Ability  to  design  and  conduct 
experiments? 

b.  ability  to  design  and  conduct 
experiments,  analyze  and 
interpret  data 

Ability  to  use  techniques,  skills 
and  tools  in  engineering  practice? 

k.  ability  to  use  the 
techniques,  skills  and  modern 
engineering  tools  necessary  for 
engineering  practice 

Expectation  to  be  current  in 
technologies  related  to  your  current 
field? 

j .  a  knowledge  of  contemporary 
issues 

Ability  to  prepare  reports  and 
documents? 

g.  ability  to  communicate 
effectively 

Ability  to  prepare  and  deliver 
professional  presentations? 

g.  ability  to  communicate 
effectively 

Ability  to  function  on  multi¬ 
disciplinary  teams? 

d.  an  ability  to  function  on 
multi-disciplinary  teams 

Understanding  of  ethical  and 
professional  responsibility? 

f.  an  understanding  of 
professional  and  ethical 
responsibility 
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Survey  Question 

Addresses  which  ABET  Criterion  3 
"Program  Outcome" 

h.  broad  education  needed  to 
understand  impact  of  engineering 
solutions  in  global  /  societal  / 
economic  context 

Recognition  of  the  need  to  engage  in 
life-long  learning? 

h.  broad  education  needed  to 
understand  impact  of  engineering 
solutions  in  global  /  societal  / 
economic  context 

i.  recognition  of  need  to  engage 
in  life  -  long  learning 

For  each  of  the  24  questions  in  the  second  section  of 


the  survey,  a  five-point  Likert  scale  us  utilized.  The 
Likert  scale  values  and  corresponding  meanings  are  clarified 
below  in  Table  3: 


Table  3:  Likert  Scale  Values  and  Corresponding  Meanings 


Number  in  Likert 
Scale 

Corresponding 
Meaning  for 
'Preparedness' 
questions 

Corresponding 
Meaning  for 
'Applicability' 
questions 

1 

Not  prepared  at  All 

Not  Important  at  all 

2 

Somewhat  Prepared 

Somewhat  Important 

3 

Adequately  Prepared 

Fairly  Important 

4 

Very  well  Prepared 

Very  Important 

5 

Extremely  well 
Prepared 

Extremely  Important 

In  addition  to  answering  specific  questions  rating  the 
levels  of  preparedness  and  applicability  of  ABET's  program 
outcomes,  respondents  were  also  asked  several  other 
questions  in  the  second  section  of  the  survey  that  may 
provide  useful  feedback  when  evaluating  their  undergraduate 
education.  Specifically,  respondents  were  asked  to  indicate 

1.  How  well  their  education,  in  general,  prepared 
them  for  their  career. 
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2.  How  many  times  per  year  they  attend  professional 
conferences . 

3.  Whether  they  have  attained,  or  intend  to  attain 
advanced  engineering  licensing  (PE  licensing) . 

4.  How  well  prepared  they  felt  for  graduate 
education . 

5.  The  highest  level  of  education  that  they  had 
attained . 

The  third  and  final  section  of  the  survey  consists  of 
three  guestions  that  allowed  respondents  to  provide 
qualitative  feedback  about  their  engineering  majors  program. 
Each  respondent  was  asked  to  provide  comments  to  identify 
and  discuss  the  most  useful  and  the  least  useful  aspects  of 
their  engineering  education  at  the  Naval  Academy.  They  were 
also  asked  to  provide  comments  or  suggestions  for 
improvement  of  the  engineering  program  at  the  Naval  Academy. 

D .  PROCEDURE 

The  design  of  the  survey  instrument  used  for  this  study 
was  the  result  of  a  collaborative  effort  between  the 
Institutional  Research,  Planning  and  Assessment  (IR) 
Department  and  the  Engineering  Department  faculty  at  the 
U.S.  Naval  Academy.  The  IR  Department,  founded  in  1992,  was 
designed  "for  the  purpose  of  evaluating  and  disseminating 
institutional  data  to  stimulate  positive  changes  in  the 
admissions  and  education  processes  at  USNA.  IR  is  the 
single  source  of  evaluated  information  on  Midshipman  and 
graduate  performance"  (USNA,  2007). 
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As  benchmarks  for  the  development  of  this  survey,  IR 
and  Engineering  Department  personnel  used  survey  instruments 
from  several  other  colleges  and  universities  that  had 
recently  completed  ABET  accreditation  visits.  These 
institutions,  including  York  College,  Johns  Hopkins 
University  and  Rutgers  University,  had  utilized  survey 
instruments  in  order  to  address  ABET's  new  accreditation 
standards,  namely  Criterion  3  "Student  Learning  Outcomes." 

The  survey  developed  by  USNA  (titled  the  "Graduate 
Performance  Assessment  Survey")  was  web-based.  The  IR 
Department  obtained  e-mail  addresses  from  the  Naval  Academy 
Alumni  Association  for  any  graduates  between  the  years  of 
1985-2005  who  had  majored  in  engineering.  Of  the  7572 
graduates  who  fit  this  description,  the  Alumni  Association 
had  contact  information  for  4189  of  them,  as  maintaining 
contact  with  the  Alumni  Association  is  voluntary  for  all 
graduates.  (This  translates  into  contact  information  for 
55.3%  of  the  stated  population.)  The  e-mail  sent  to  each 
graduate  contained  the  Website  URL  where  the  survey  was 
located;  respondents  simply  needed  to  'click'  on  the  URL, 
open  the  survey  and  begin  entering  data. 

The  IR  Department  was  responsible  for  survey 
administration  and  data  integrity.  Following  a  brief  pilot 
period,  which  was  conducted  in  order  to  determine  if  the 
survey  was  complete,  accessible  and  user-friendly,  the 
survey  was  placed  online  for  a  period  of  one  month  between 
February  and  March  of  2006.  After  the  survey  was  taken 
offline,  the  IR  Department  pulled  data  from  the  electronic 
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responses  and  dropped  them  into  a  spreadsheet,  and  cleaned 
the  data.  The  complete  survey  instrument  is  located  in 
Appendix  A. 

E .  SAMPLE 

The  sample  for  this  study  is  a  total  of  1068  Naval 
Academy  graduates  who  majored  in  engineering  between  1985  - 
2005.  Of  the  4189  graduates  from  this  time  period  for  whom 
contact  information  was  available  and  to  whom  the  survey  was 
sent,  the  1068  respondents  yield  a  response  rate  of  25.5%. 


Within  the  entire  population  of  7572  engineering  majors  from 
this  time  period,  the  1068  respondents  represent  14.1%  of 
the  population. 

All  of  the  participants  in  this  study  were  required  to 
indicate  their  specific  field  of  engineering  study.  The 
frequencies  for  each  of  the  eight  majors  within  USNA' s 
Engineering  Department  are  listed  below  in  Table  4.  Of  these 
majors,  all  are  still  available  for  study  at  the  Naval 
Academy  with  the  exception  of  Marine  Engineering,  which  was 
last  offered  in  1999. 


Table  4 :  Ma j  or 


Engineering 

Majors 

Number  in 
Population 

Percent  of 
Population 

Number  in 
Sample 

Percent  of 
Sample 

Aerospace 

(EAS) 

838 

11 . 10% 

106 

9.90% 

Aeronautics 

(EASA) 

505 

6.70% 

82 

7.70% 

Electrical 
(EEE ) 

590 

7 . 80% 

92 

8 . 60% 
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Engineering 

Majors 

Number  in 
Population 

Percent  of 
Population 

Number  in 
Sample 

Percent  of 
Sample 

General  (EGE) 

1020 

13.50% 

100 

9.40% 

Mechanical 

(EME) 

1376 

18.20% 

210 

19.60% 

Naval 

Architecture 

(ENA) 

431 

5.70% 

63 

5.90% 

Ocean 

Engineering 
(EOE ) 

873 

11.50% 

111 

10.40% 

Systems  (ESE) 

1684 

22.20% 

246 

23.00% 

Marine 

Engineering 

(ESP) 

255 

3.40% 

58 

5.40% 

Total 

7572 

100% 

1068 

100% 

When  comparing  the  population  to  the  sample,  all 
engineering  majors,  with  the  exception  of  aerospace,  ocean 
and  general  engineering,  are  slightly  over-represented  in 
the  sample.  The  most  over-represented  major  is  marine 
engineering,  where  2.0%  more  of  the  sample  is  comprised  of 
marine  engineers  than  is  the  case  within  the  population. 
Conversely,  the  most  under-represented  major  is  general 
engineering;  13.5%  of  the  population  majored  in  general 
engineering,  compared  with  only  9.4%  of  the  sample  (a 
difference  of  4.1%)  .  With  the  exception  of  general 
engineering,  the  samples  of  all  engineering  majors  represent 
the  population  within  a  tolerance  of  2.0%  or  better. 

For  purposes  of  this  study,  there  is  a  need  to  track  as 
much  specific  biographical  information  about  each  of  our 
respondents  as  possible  in  order  to  maximize  the 
opportunities  for  meaningful  comparisons  between  distinct 
groups.  For  example,  one  focus  of  this  study  is  on  whether 
graduates  are  currently  in  an  active  duty  military  status  or 
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a  civilian  status,  and  also  which  particular  field  of 
engineering  each  graduate  majored  in  while  attending  the 
Naval  Academy.  For  a  list  of  demographic  variables  and 
descriptive  statistics,  see  Appendix  B. 

Tracking  and  identification  information  is  made 
possible  by  having  respondents  indicate  their  alpha  code. 
The  alpha  code  is  a  unique  identification  number  issued  to 
Midshipmen  on  the  day  they  report  to  the  Naval  Academy, 
which  they  maintain  until  graduation.  Alpha  codes  are  not 
duplicated  or  reissued  regardless  of  graduation  year. 


Table  5 :  Graduation  Year 


Grad  Year 

Sample 

Percent  of  Sample 

1985  -  1990 

321 

30.10% 

1991  -  1995 

272 

25.50% 

1996  -  2000 

270 

25.30% 

2001  -  2005 

205 

19.20% 

Total 

1068 

100% 

The  graduating  classes  of  1985-1990  are  slightly  over¬ 
represented  in  our  sample,  whereas  the  graduating  classes  of 
2001-2005  are  slightly  under-represented.  There  is  a  good 
sample  distribution  across  the  graduating  classes  of  1991- 
2000,  which  represents  approximately  50%  of  the  sample. 


Table  6:  Civilian  /  Military  Status 


Status 

Frequency 

Percent  of 
Sample 

Civilian 

546 

51 . 1 

Active  Duty  Military 

522 

48 . 9 

Total 

1068 

100.0 
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This  is  close  to  an  ideal  sample  distribution  between 
civilian  and  military  respondents,  as  a  good  bit  of  analysis 
is  conducted  with  the  intent  of  comparing  civilian  to  active 
duty  military  respondents.  The  sample  distribution  is 
roughly  equal,  with  civilian  and  active  duty  military  both 
representing  about  half  of  the  sample. 

With  few  exceptions,  all  graduates  of  the  Naval  Academy 
are  required  to  accept  commissions  as  either  Navy  or  Marine 
Corps  officers  following  graduation.  Therefore,  all 
respondents  served  at  least  one  year  on  active  duty.  As  a 
general  rule,  graduates  will  owe  a  minimum  of  5  years  on 
active  duty,  although  this  number  frequently  changes  based 
on  many  factors  including  warfare  community,  needs  of  the 
service,  and  health  status  of  the  service  member.  All 
respondents  indicated  the  total  amount  of  time  they  spent  on 
active  duty,  which  ranged  in  time  from  1  year  to  8  years.  In 
the  sample,  90.1%  of  the  respondents  served  between  3  and  5 
years  on  active  duty. 

Respondents  are  also  asked  to  indicate  the  warfare 
community  in  which  they  served  (or  serve) .  The  vast  majority 
of  the  sample  (81.5%)  indicated  that  they  served  in  one  of 
the  four  largest  unrestricted  line  warfare  communities  that 
are  available  to  Naval  Academy  graduates,  namely  naval 
aviation,  surface  warfare,  submarine  warfare  and  the  Marine 
Corps.  Approximately  9%  of  respondents  serve  (or  served)  in 
the  Civil  Engineering  Corps  or  as  Engineering  Duty  Officers. 

Survey  respondents  were  asked  to  answer  questions  in 
order  to  provide  details  as  to  the  nature  of  their  current 
job.  For  example,  out  of  515  active  duty  military 
respondents,  69.3%  indicated  that  the  nature  of  their 
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current  job  is  technical,  with  the  remaining  30.7% 
indicating  that  their  current  job  is  non-technical . 

In  contrast,  57.0%  of  civilian  respondents  indicated 
that  their  current  occupation  is  technical  in  nature,  and 
43.0%  indicated  that  their  current  occupations  are  non¬ 
technical  in  nature.  (To  see  further  information  concerning 
this  breakdown  between  technical  and  non-technical 
positions,  see  Appendix  B.) 

Civilian  respondents  were  asked  to  indicate  the  level 
of  their  current  position  within  their  current  organization, 
ranging  from  entry  level  to  senior  executive  (i.e.. 
President,  CEO) .  Again,  a  complete  description  of  this 
breakdown  can  be  found  in  Appendix  B.  Of  the  civilian 
respondents,  7.4%  indicated  that  they  are  the  Senior 
Executive  of  their  organization;  40.5%  indicated  that  they 
work  as  upper  management,  and  43.8%  indicated  that  they  work 
as  mid-level  management.  A  small  percentage  (3.3%)  indicated 
that  they  are  currently  working  at  entry  level  positions. 

F.  ANALYSIS 

The  majority  of  the  analysis  for  this  study  focuses  on 
the  entire  population,  with  an  effort  being  made  to 
determine  the  level  of  applicability  and  preparedness  that 
respondents  assign  to  each  of  the  engineering  related  skills 
and  abilities.  Additional  analysis  focuses  on  comparing 
military  and  civilian  respondents,  and  also  on  comparing 
respondents  from  differing  engineering  majors.  The  portion 
of  the  survey  instrument  that  is  utilized  for  comparison 
analyses  are  the  guestions,  described  earlier  in  this 
chapter,  modeled  after  ABET''  s  "Criterion  3"  (a  -  k)  .  These 
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questions  focus  on  specific  student  learning  outcomes,  and 
allow  respondents  to  comment  on  levels  of  applicability  of 
certain  skills  to  their  profession  and  adequacy  of 
preparation  attained  during  undergraduate  study. 

Independent  samples  T-tests  are  conducted  in  order  to 
compare  military  and  civilian  respondents.  The  goal  is  to 
identify  statistically  significant  differences  between 
military  and  civilian  respondents  with  respect  to  how  they 
view  the  applicability  of  certain  engineering  related  skills 
to  their  jobs,  and  also  to  identify  significant  differences 
in  levels  of  preparation.  All  significance  criteria  (for  all 
analyses)  will  be  p  <  .05. 

For  both  civilian  and  military  respondents,  paired 
sample  T-tests  are  conducted  in  order  to  determine  if  there 
are  statistically  significant  differences  between  the  levels 
of  preparation  and  the  levels  of  applicability  for  each  of 
the  stated  engineering  skills.  In  other  words,  the  paired 
sample  test  is  used  to  determine  whether  respondents  within 
each  of  the  two  groups  (military  or  civilian)  believed  that 
their  level  of  preparedness  for  each  engineering  skill  was 
high  or  low  relative  to  the  perceived  level  of  skill 
applicability . 

Analysis  of  variance  (ANOVA)  tests  are  conducted  in 
order  to  compare  respondents  across  the  nine  different 
engineering  majors.  Specifically,  the  ANOVA  test  is  utilized 
to  determine  if  there  is  a  statistically  significant 
difference  between  respondents  of  different  engineering 
majors  with  regard  to  how  well  they  feel  they  were  prepared 
in  engineering  related  skills,  or  how  applicable  they 
believe  those  skills  to  be. 
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IV.  DATA  ANALYSIS 


A.  ENTIRE  SAMPLE:  APPLICABILITY  VS.  PREPAREDNESS 

All  respondents  were  asked  to  indicate  the  extent  to 
which  they  felt  that  each  of  12  different  engineering 
related  skills  were  applicable  to  their  current  profession. 
They  were  also  asked  to  indicate  the  extent  to  which  they 
felt  that  their  undergraduate  engineering  education  had 
prepared  them  to  perform  these  engineering  related  skills. 
Answers  to  these  questions  ranged  from  1  (  'Not  important  at 

all'  or  '  Not  prepared  at  all' )  to  5  (  'Extremely  important' 

or  'Extremely  well  prepared'  )  .  The  results  for  the  entire 
sample  are  summarized  in  Table  7. 

Table  7:  Sample  Mean  Values  for  Applicability  and 

Preparedness 


Engineering  Skill  or  Ability 

Applicability 

Preparedness 

Mean 

N 

Mean 

N 

Apply  knowledge  of  math,  engineering  and 
science . 

3.84 

1037 

4.39 

1065 

Ability  to  analyze  and  interpret  data. 

4.35 

1015 

4.48 

1066 

Identify,  formulate,  and  solve  engineering 
problems . 

3.09 

978 

4.27 

1065 

Design  systems,  components,  or  processes 
to  meet  needs . 

2 . 97 

980 

3.79 

1063 

Design  and  conduct  experiments. 

2.34 

934 

3.78 

1062 

Use  techniques,  skills,  and  tools  in 
engineering  practice. 

3.03 

962 

4.07 

1060 

Expectation  to  be  current  in  technologies 
related  to  your  career  field. 

3.89 

1001 

3.74 

1060 

Prepare  reports  and  documents. 

4.38 

991 

4.26 

1063 

Prepare  and  deliver  professional 
presentations . 

4.48 

992 

4 .11 

1062 

Function  on  multi-disciplinary  teams. 

4.4 

982 

4.35 

1062 

Understanding  of  ethical  and  professional 
responsibility . 

4.74 

964 

4 . 67 

1064 

Recognition  of  the  need  to  engage  in  life¬ 
long  learning. 

4.44 

988 

4.38 

1060 
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The  six  skills  reported  to  be  most  applicable  (means  > 
4.3)  to  the  respondents'  current  jobs  were:  "ability  to 
analyze  and  interpret  data;"  "prepare  reports  and 
documents;"  "prepare  and  deliver  professional 
presentations;"  "function  on  multi-disciplinary  teams;" 
"understanding  of  ethical  and  professional  responsibility;" 
and  "recognition  of  the  need  to  engage  in  life-long 
learning."  Four  skills  had  mean  ratings  at  or  below  the 
mid-point  rating  of  3  for  applicability:  "Identify, 
formulate,  and  solve  engineering  problems;"  "design  systems, 
components,  or  processes  to  meet  needs;"  "design  and  conduct 
experiments;"  and  "use  technigues,  skills,  and  tools  in 
engineering  practice." 

All  but  three  skill  areas  received  ratings  >4.0  in 
terms  of  how  well  the  degree  program  prepared  them.  The 
highest  rated  skill  area  was  "understanding  of  ethical  and 
professional  responsibility"  (mean  =  4.67).  The  three  skill 
areas  with  the  lowest  ratings  of  preparation  were:  "design 
systems,  components,  or  processes  to  meet  needs";  "design 
and  conduct  experiments";  and  "expectation  to  be  current  in 
technologies  related  to  your  current  field."  It  is 
noteworthy  that  the  first  two  of  these  lowest-rated  skills 
also  receive  the  lowest  ratings  of  applicability  to  their 
current  jobs. 

For  6  of  the  12  skills,  respondents  indicated,  on 
average,  that  their  level  of  preparation  met  or  exceeded  the 
level  of  ability  required  in  that  skill  area  in  their 
current  job  or  profession.  Notably,  respondents  felt  well- 
prepared  to  identify,  formulate  and  solve  engineering 
problems;  design  systems,  components,  or  processes  to  meet 
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needs;  design  and  conduct  experiments;  and  use  techniques, 
skills,  and  tools  in  engineering  practice,  yet  they  did  not 
view  these  skills  as  highly  relevant  to  their  current 
positions . 

For  the  other  six  skills,  respondents  indicated,  on 
average,  that  their  level  of  preparation  did  not  meet  the 
level  of  ability  required  in  that  skill  area.  Notably, 
respondents  felt  under-prepared  with  regard  to  their  ability 
to  prepare  reports  and  documents,  and  to  prepare  and  deliver 
professional  presentations. 

B.  EFFECTS  OF  APPLICABILITY  ON  PREPAREDNESS 

The  intention  of  this  section  is  to  examine  the  results 
only  for  the  survey  respondents  who  believe  that  these  12 
engineering  related  skills  are  applicable  to  their  current 
job  or  profession.  For  those  respondents  who  reported  that 
the  skills  are  applicable,  levels  of  perceived  preparedness 
were  analyzed.  Those  respondents  who  indicated  that  a  given 
skill  was  "not  important  at  all"  or  "somewhat  important" 
were  eliminated  from  the  analysis  of  each  skill  category. 
Only  those  respondents  who  indicated  that  a  given  skill  was 
"fairly  important, "  "very  important"  or  "extremely 
important"  were  included  in  this  analysis.  The  results  are 
summarized  in  Table  8. 
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Table  8:  How  Well  USNA  Degree  Prepared  Graduates  in  Skill 
Areas  Applicable  to  Current  Work 


Engineering  Skill  or  Ability 

PREPAREDNESS 

MEAN 

N 

Apply  knowledge  of  math,  engineering  and  science. 

4.42 

893 

Ability  to  analyze  and  interpret  data. 

4.48 

944 

Identify,  formulate,  and  solve  engineering  problems. 

4.35 

616 

Design  systems,  components,  or  processes  to  meet  needs. 

3.94 

563 

Design  and  conduct  experiments. 

3.92 

366 

Use  techniques,  skills,  and  tools  in  engineering 
practice . 

4 .19 

567 

Expectation  to  be  current  in  technologies  related  to 
your  career  field. 

3.77 

841 

Prepare  reports  and  documents. 

4.28 

926 

Prepare  and  deliver  professional  presentations. 

4 . 1 

939 

Function  on  multi-disciplinary  teams. 

4.36 

914 

Understanding  of  ethical  and  professional 
responsibility . 

4 . 67 

943 

Recognition  of  the  need  to  engage  in  life-long  learning. 

4.38 

946 

All  skill  categories  have  preparedness  ratings  greater 
than  3.75,  and  all  but  three  are  greater  than  4.0.  For  6  of 
the  12  skill  categories,  more  than  900  respondents  indicated 
high  levels  of  applicability:  'ability  to  analyze  and 
interpret  data;"  "prepare  reports  and  documents;"  "prepare 
and  deliver  professional  presentations;"  function  on  multi¬ 
disciplinary  teams;"  "understanding  of  ethical  and 
professional  responsibility;"  and  "recognition  of  the  need 
to  engage  in  life-long  learning."  Of  these  six  skills,  the 
abilities  to  "prepare  reports  and  documents"  and  "prepare 
and  deliver  professional  presentations"  received  the  lowest 
ratings  for  preparedness. 

The  'N'  values  themselves  are  telling  in  this  analysis. 
Approximately  20%  to  50%  fewer  respondents  indicated  high 
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levels  of  applicability  for  four  of  the  12  skill  areas  (as 
compared  with  the  other  skill  categories) :  "identify, 
formulate,  and  solve  engineering  problems;"  "design  systems, 
components,  or  processes  to  meet  needs;"  "design  and  conduct 
experiments;"  and  "use  techniques,  skills,  and  tools  in 
engineering  practice."  But  for  those  who  did  feel  that  these 
skills  were  applicable,  their  perceived  levels  of 
preparedness  are  higher  than  the  overall  sample  means 
reported  in  the  previous  table. 

It  is  also  interesting  to  note  that,  in  general,  as  the 
number  of  respondents  who  believed  that  a  given  skill  was 
applicable  to  their  current  profession  increases,  the 
average  indication  of  preparedness  for  that  skill  also 
increases.  The  largest  number  of  respondents  indicated  that 
the  skills  "understanding  of  ethical  and  professional 
responsibility;"  "ability  to  analyze  and  interpret  data"  and 
"recognition  of  the  need  to  engage  in  life-long  learning" 
were  applicable  to  their  current  jobs,  and  these  skills  were 
also  rated  highest  in  terms  of  preparedness. 

C.  PAIRED  DIFFERENCES  FOR  ENGINEERING  SKILLS 

In  this  section,  each  of  the  12  "preparedness" 
questions  was  paired  with  their  corresponding 
"applicability"  questions.  A  paired  samples  t-test  was 
conducted  in  order  to  determine  any  significant  difference 
between  the  level  of  preparedness  and  the  level  of 
applicability  for  each  of  the  engineering  related  skills. 
The  results  of  the  paired  samples  t-test  (using  the  entire 
sample)  are  shown  in  Table  9. 
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Table  9:  Mean  Differences  Between  'Preparedness'  and 

'Applicability' 


Engineering  Skill  or  Ability 

N 

Mean  Diff  ( 'Preparedness' 

-  'Applicability' ) 

Sig . 

Apply  knowledge  of  math, 
engineering  and  science. 

1062 

0.49 

0.000 

Ability  to  analyze  and  interpret 
data . 

1060 

0.06 

0.099 

Identify,  formulate,  and  solve 
engineering  problems. 

1057 

0.96 

0.000 

Design  systems,  components,  or 
processes  to  meet  needs. 

1058 

0.60 

0.000 

Design  and  conduct  experiments. 

1057 

1.00 

0.000 

Use  techniques,  skills,  and  tools 
in  engineering  practice. 

1054 

0.77 

0.000 

Expectation  to  be  current  in 
technologies  related  to  your 
career  field. 

1052 

-0.26 

0.000 

Prepare  reports  and  documents. 

1057 

-0.22 

0.000 

Prepare  and  deliver  professional 
presentations . 

1058 

-0.47 

0.000 

Function  on  multi-disciplinary 
teams . 

1056 

-0.17 

0.000 

Understanding  of  ethical  and 
professional  responsibility. 

1056 

-0.18 

0.000 

Recognition  of  the  need  to  engage 
in  life-long  learning. 

1054 

-0.17 

0.000 

A  positive  value  for  the  mean  difference  indicates  that 
respondents,  on  average,  gave  higher  ratings  to  their 
perceived  level  of  preparedness  relative  to  the  rating  of 
applicability.  Conversely,  a  negative  value  for  the  mean 
difference  indicates  that  respondents,  on  average,  gave 
lower  ratings  to  their  perceived  level  of  preparedness 
relative  to  the  rating  of  applicability  for  that  skill. 


For 

the 

skills  listed  below. 

the  mean 

difference 

between 

the 

level  of  preparedness 

and 

the 

level 

of 

applicabi 

lity 

was  positive  and  statistically 

significant 

(P 

<  .05)  .  That  means  that  respondents  (on  average)  gave  high 
ratings  for  preparedness  in  these  skill  areas  relative  to 
the  level  of  skill  applicability: 
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1.  Ability  to  apply  knowledge  of  math,  engineering  and 
science . 

2.  Ability  to  identify,  formulate  and  solve  engineering 
problems . 

3.  Ability  to  design  systems,  components  or  processes 
to  meet  needs . 

4.  Ability  to  design  and  conduct  experiments. 

5.  Ability  to  use  techniques,  skills  and  tools  in 
engineering  practice. 

For  the  skills  listed  below,  the  mean  difference 
between  the  level  of  preparedness  and  the  level  of 
applicability  was  negative  and  statistically  significant  (p 
<  .05),  meaning  that  respondents  (on  average)  indicated  low 
ratings  for  preparedness  in  these  skill  areas  relative  to 
the  level  of  skill  applicability: 

1.  Expectation  to  be  current  in  technologies  related  to 
your  current  field. 

2.  Ability  to  prepare  reports  and  documents. 

3.  Ability  to  prepare  and  deliver  professional 
presentations . 

4.  Ability  to  function  on  multi-disciplinary  teams. 

5.  Understanding  of  ethical  and  professional 
responsibility . 

6.  Recognition  of  the  need  to  engage  in  life-long 
learning . 
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D.  EFFECTS  OF  JOB  TYPE  ON  PERCEIVED  PREPAREDNESS 

All  respondents  were  asked  the  general  question,  "How 
well  did  your  engineering  major  at  USNA  prepare  you  for  your 
current  career?"  Responses  ranged  from  1  (  'Not  Prepared  at 

All'  )  to  5  (  'Extremely  Well  Prepared'’  )  .  For  this  section,  an 
independent  samples  T-test  was  conducted  in  order  to 
determine  whether  respondents  who  reported  that  their 
current  job  is  technical  in  nature  believed  that  they  are 
better  prepared  than  those  who  reported  that  their  current 
job  is  non-technical  in  nature.  The  grouping  variable  is  a 
binomial  variable  that  indicates  whether  a  respondent's  job 
is  technical  or  non-technical,  and  the  test  variable  is  the 
respondents'  rating  of  their  level  of  preparedness.  The 
results  are  summarized  in  Table  10. 

Table  10:  How  Well  did  Major  Prepare  for  Current  Career? 

(Technical  vs.  Non-Technical ) 


Job  Description 

N 

Mean 

Std.  Dev 

Technical 

664 

4 . 05 

0.867 

Non -Technical 

388 

3.79 

1.019 

The  overall  sample  mean  for  this  'preparedness' 
variable  is  3.95.  The  results  of  the  t-test  indicate  that 
there  is,  in  fact,  a  statistically  significant  difference 
(F=25.51,  p  <  .05)  between  respondents  in  technical  jobs  and 
those  in  non-technical  jobs  with  regard  to  how  they  perceive 
their  level  of  preparation.  In  particular,  respondents  who 
are  in  technical  jobs  give  higher  ratings  to  their  level  of 
preparation  than  those  who  are  in  non-technical  jobs. 
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E.  EFFECTS  OF  MAJOR  ON  PERCEIVED  PREPAREDNESS 

Respondents  were  asked  the  general  question,  "How  well 
did  your  engineering  major  at  USNA  prepare  you  for  your 
current  career?"  Responses  ranged  from  1  ("Not  Prepared  at 
All")  to  5  ("Extremely  Well  Prepared") .  A  one-way  analysis 
of  variance  (ANOVA)  test  was  conducted,  using  the  major  as 
the  grouping  variable  and  level  of  preparedness  as  the  test 
variable.  This  test  was  conducted  to  determine  if  there  was 
a  significant  difference  in  the  means  between  each  group. 
The  results  are  summarized  in  Table  11. 


Table  11:  How  Well  did  Major  Prepare  for  Current  Career? 


Major 

N 

Mean 

Value 

Std . 

Dev . 

Systems  Engineering 

245 

4 

.00 

0. 

896 

Mechanical  Engineering 

210 

3 

.  90 

0. 

890 

Ocean  Engineering 

111 

3 

.  96 

0. 

972 

Aerospace  Engineering 

104 

3 

.  91 

0. 

946 

General  Engineering 

100 

3 

.75 

0. 

925 

Electrical  Engineering 

92 

4 

.09 

0. 

898 

Aeronautics  Engineering 

82 

4 

.15 

0. 

904 

Naval  Architecture 

62 

3 

.  95 

1 . 

093 

Marine  Engineering 

58 

3 

.86 

0. 

999 

Total 

1064 

3 

.95 

0. 

932 

Homogeneity  of  variance  between  the  major  groups  was 
verified  to  be  true,  and  the  results  of  the  ANOVA  test 
indicate  that  there  is  no  statistically  significant 
difference  (F=1.55,  p  <  .05)  between  the  major  groups  with 

respect  to  how  prepared,  in  general,  each  group  felt. 

F.  PREPARATION  FOR  ADVANCED  EDUCATION 

Respondents  were  asked  to  indicate  whether  or  not  they 
had  attained  any  forms  of  advanced  education  during  the  time 
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since  their  graduation  from  the  Naval  Academy,  ranging  from 
partial  completion  of  a  graduate  degree  to  the  completion  of 
a  Doctorate  degree.  They  were  also  asked  to  indicate  the 
extent  to  which  they  felt  that  their  undergraduate  education 
had  prepared  them  for  graduate  education.  (This  question  was 
posed  to  all  respondents,  whether  they  completed  further 
education  or  not.)  To  indicate  their  level  of  preparation, 
respondents  chose  a  number  between  1  ( 'not  at  all  prepared' ) 
and  5  (  'extremely  well  prepared' )  .  The  descriptive 
statistics  for  this  analysis  are  shown  in  Table  12  below. 
The  first  column  indicates  the  highest  level  of  education 
attained,  and  the  mean  value  indicates  the  extent  to  which 
that  group  felt  prepared  for  graduate  education. 


Table  12:  Feeling  of  Preparedness  for  Advanced  Education 


EDUCATION  LEVEL 

N 

MEAN 

STD.  DEV. 

PhD  or  equivalent 

29 

4.03 

0.906 

Master's  Degree  or  equivalent 

515 

4.06 

0.914 

Some  graduate  work 

114 

3.98 

0.902 

Bachelor's  Degree 

393 

3.74 

0.803 

A  one-way  analysis  of  variance  (ANOVA)  test  was 
conducted  in  order  to  determine  whether  a  significant 
difference  existed  between  group  means.  The  grouping 
variable  was  the  highest  level  of  education  attained,  and 
the  test  variable  was  the  level  of  preparedness  for  graduate 
education.  The  results  of  this  test  indicate  that  there  are 
significant  differences  (F=9.89,  p  <  .05)  between  at  least 
two  of  these  group  means. 

In  order  to  determine  which  groups  differ,  a  post  hoc 
LSD  test  was  conducted.  The  results  indicate  that  those 
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respondents  with  a  master' s  degree  felt  significantly  more 
prepared  for  graduate  education  than  those  respondents  who 
have  not  furthered  their  education  beyond  a  bachelor' s 
degree  (p  <  .05) .  Additionally,  those  respondents  who  have 
completed  some  graduate  work  felt  significantly  more 
prepared  for  graduate  education  than  those  respondents  who 
have  not  furthered  their  education  beyond  their  Bachelor' s 
degree  (p<  .05) .  These  results  indicate  that  those  who 

pursue  graduate  education  have  a  higher  perceived 
preparation  than  those  who  have  only  completed  the 
bachelor's  degree. 

G.  RELATIONSHIP  BETWEEN  CAREER  AND  EDUCATION 

Civilian  respondents  were  asked  to  indicate  the  extent 
to  which  their  current  job  is  related  to  their  undergraduate 
engineering  education  at  the  Naval  Academy.  Possible  answers 
were  'Not  At  All,'  'Somewhat'  or  'To  a  Great  Extent.'  For 
the  purposes  of  this  specific  analysis,  these  responses  were 
recoded  into  a  new  binomial  numeric  variable,  in  which  those 
who  answered  'Not  At  All'  are  classified  as  having  no 
relationship  between  undergraduate  education  and  job,  and 
those  who  answered  'Somewhat'  or  'To  a  Great  Extent'  are 
classified  as  having  a  relationship  between  undergraduate 
education  and  job. 

All  respondents  were  asked  the  general  question,  "How 

well  did  your  engineering  major  at  USNA  prepare  you  for  your 

current  career?"  Responses  ranged  from  1  ("Not  Prepared  at 

All")  to  5  ("Extremely  Well  Prepared") .  An  independent 

samples  t-test  was  conducted  in  order  to  determine  whether 

those  civilian  respondents  who  felt  that  their  education 

applied  to  their  current  job  felt  more  or  less  prepared  than 
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those  respondents  who  did  not  see  any  relationship  between 
their  education  and  their  job.  The  descriptive  statistics 
for  this  analysis  are  displayed  in  Table  13. 


Table  13:  "How  Well  did  Major  Prepare  for  Current  Career" 

(Civilians  Only) 


Job  Related  to  Education? 

N 

Mean 

Related 

346 

4.06 

Not  Related 

195 

3.62 

The  results  of  the  t-test  indicate  that  there  is  a 
significant  difference  (t=-4.92,  p  <  .05)  between  these  two 
groups.  Here,  those  civilian  respondents  who  believed  that 
their  job  was  related  to  their  engineering  education  felt 
significantly  better  prepared  for  their  career  than  those 
respondents  who  report  no  relationship  between  job  and 
education . 

H.  CIVILIAN/MILITARY:  INDEPENDENT  SAMPLE  T-TEST  RESULTS 

1.  Overall  Feeling  of  Preparedness 

Respondents  were  asked  the  general  question,  "How  well 
did  your  engineering  major  at  USNA  prepare  you  for  your 
current  career?"  Responses  ranged  from  1  ("Not  Prepared  at 
All")  to  5  ("Extremely  Well  Prepared") .  An  independent 
samples  t-test  was  conducted,  using  a  binomial  grouping 
variable  that  indicates  whether  respondents  are  civilian  or 
military,  and  overall  level  of  preparedness  as  the  test 
variable.  This  test  was  conducted  to  determine  if  there  was 
a  significant  difference  in  the  means  for  the  civilian  and 
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military  groups,  with  respect  to  their  level  of  perceived 
overall  preparedness.  The  results  are  presented  in  Table  14. 


Table  14:  How  Well  did  Major  Prepare  for  Current  Career? 

(Civilian  /  Military) 


Status 

N 

Mean 

Std.  Dev. 

Civilian 

543 

3.90 

0.963 

Active  Duty  Military 

521 

4 .01 

0.895 

The  significance  level  of  the  t-test  was  0.065.  Thus, 
it  cannot  be  concluded  that  there  is  a  significant 
difference  (p  <  .05)  between  the  two  group  means  (civilian 
and  military) ,  although  there  is  a  significant  difference  at 
the  P  <  .10  level,  with  military  rating  themselves  as  better 
prepared  than  their  civilian  employed  counterparts. 

2.  Preparedness  in  Applicable  Engineering  Related 
Skills 

An  independent  samples  t-test  was  conducted,  using  a 
binomial  grouping  variable  that  indicates  whether 
respondents  are  civilian  or  active  duty  military.  The  test 
variables  were  the  12  "preparedness"  guestions  that  allowed 
respondents  to  indicate  their  level  of  preparedness  with 
respect  to  twelve  different  engineering  related  skills. 

The  intention  of  this  analysis  is  to  examine  the 
responses  provided  by  survey  respondents  who  believe  that 
these  engineering  related  skills  are  applicable  to  their 
current  job  or  profession.  With  this  in  mind,  those 
respondents  who  indicated  that  a  given  skill  was  "not 
important  at  all"  or  "somewhat  important"  were  eliminated 
from  the  analysis  of  each  skill  category  prior  to  each 
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independent  sample  test.  Only  those  respondents  who 
indicated  that  a  given  skill  was  "fairly  important, "  "very 
important"  or  "extremely  important"  were  included  in  this 
analysis . 

The  results  of  this  analysis  are  summarized  in  Table 
15.  For  9  of  the  12  engineering  related  skills,  there  was  no 
statistically  significant  difference  (p  <  .05)  between 
military  and  civilian  with  respect  to  preparedness.  For  the 
ability  to  "apply  knowledge  of  math  and  science;"  "function 
on  multi-disciplinary  teams;"  and  "understand  ethical  and 
professional  responsibility,"  civilians  indicated  a  higher 
level  of  preparedness  (p  <  .05) . 


Table  15:  How  well  USNA  Degree  Prepared  Graduates  in  Skill 
Areas  Applicable  to  Current  Profession 


Engineering  Skill  or 
Ability 

Civilian 

Military 

t-test  sig. 

Mean 

N 

Mean 

N 

Apply  knowledge  of  math, 
engineering  and  science. 

4.48 

469 

4 . 35 

424 

.002 

Ability  to  analyze  and 
interpret  data. 

4 . 52 

494 

4.44 

450 

.  072 

Identify,  formulate,  and 
solve  engineering 
problems . 

4 . 37 

330 

4 . 32 

286 

.390 

Design  systems, 
components,  or  processes 
to  meet  needs . 

3.89 

334 

4  .  02 

229 

.065 

Design  and  conduct 
experiments . 

3.86 

220 

4.01 

146 

.  122 

Use  techniques,  skills, 
and  tools  in  engineering 
practice . 

4 .22 

300 

4  .  14 

267 

.213 

Expectation  to  be  current 
in  technologies  related  to 
your  career  field. 

3 .75 

433 

3.80 

408 

.458 

Prepare  reports  and 
documents . 

4.29 

470 

4.26 

456 

.470 

Prepare  and  deliver 
professional 
presentations . 

4  .  07 

473 

4  .  13 

466 

.317 

Function  on  multi¬ 
disciplinary  teams. 

4.44 

473 

4.28 

441 

.002 

Understanding  of  ethical 
and  professional 
responsibility . 

4 .74 

477 

4  .  60 

466 

.000 

Recognition  of  the  need  to 
engage  in  life-long 
learning . 

4  .  43 

480 

4 . 34 

466 

.090 

52 


3. 


Technical  vs.  Non-Technical :  Do  Civilians  Differ 
from  Military? 


In  an  earlier  section,  an  independent  samples  t-test 
was  conducted  in  order  to  determine  whether  respondents  who 
worked  in  technical  jobs  felt  differently  about  their  level 
of  preparation  than  those  who  worked  in  non-technical  jobs. 
The  results  of  that  analysis  indicated  that,  in  general, 
those  in  technical  jobs  felt  significantly  better  prepared 
(p  <  .05)  than  those  in  non-technical  jobs. 

In  this  section,  the  intention  is  to  further  this 
analysis  by  comparing  civilian  and  military  respondents.  All 
respondents  were  asked  the  general  question,  "How  well  did 
your  engineering  major  at  USNA  prepare  you  for  your  current 
career?"  Responses  ranged  from  1  ("Not  Prepared  at  All")  to 
5  ("Extremely  Well  Prepared") . 

In  the  analysis,  respondents  are  first  grouped 
according  to  whether  they  are  military  or  civilian.  Two 
independent  samples  t-tests  were  conducted  (one  for  the 
civilian  group  and  one  for  the  military  group)  in  order  to 
determine  whether  respondents  who  felt  that  their  current 
jobs  were  technical  in  nature  believed  that  they  were  better 
prepared  than  those  who  felt  that  their  current  job  is  non¬ 
technical  in  nature.  The  grouping  variable  is  a  binomial 
variable  that  indicates  whether  a  respondent's  job  is 
technical  or  non-technical,  and  the  test  variable  is  the 
respondents'  rating  of  their  level  of  preparedness.  The 
results  are  summarized  in  Tables  16  and  17. 


53 


Table  16:  How  well  did  Major  Prepare  for  Current  Career? 
(Technical  vs.  Non-technical  for  Civilian  Respondents) 


Civilian  Job  Description 

N 

Mean 

Std.  Dev. 

Technical 

308 

3.97 

0.898 

Non-Technical 

230 

3.81 

1 . 048 

Table  17:  How  well  did  Major  Prepare  for  Current  Career? 
(Technical  vs.  Non-technical  for  Military  Respondents) 


Military  Job  Description 

N 

Mean 

Std.  Dev. 

Technical 

356 

4 .11 

0.836 

Non-Technical 

158 

3 .77 

0. 977 

For  the  civilian  respondents,  the  significance  of  the 
t-test  was  0.055.  Thus,  it  cannot  be  concluded  that  a 
significant  difference  (p  <  .05)  exists  between  technical 

and  non-technical  jobs  within  the  civilian  group.  Within  the 
active  duty  military  group,  however,  the  significance  value 
was  smaller  than  .0001,  indicating  that  a  significant 
difference  does  exist  between  technical  and  non-technical 
jobs  within  the  military.  In  particular,  those  military 
respondents  who  felt  that  their  jobs  were  technical  in 
nature  believed,  on  average,  that  they  were  better  prepared 
for  their  job  due  to  their  undergraduate  engineering 
education.  In  both  cases  (civilian  and  military),  the 
average  value  for  "level  of  preparedness"  was  higher  for 
those  who  believed  their  jobs  to  be  technical,  although  the 
difference  is  statistically  significant  only  for  the 
military  group. 
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I.  CIVILIAN/MILITARY:  PAIRED  SAMPLES  T-TEST  RESULTS 

For  the  paired  samples  t-tests,  the  cases  were  split 
into  two  categories  (civilian  and  military) .  Each  of  the  12 
"preparedness"  questions  were  paired  with  their 
corresponding  "applicability"  questions,  and  the  paired 
samples  t-test  was  conducted  in  order  to  determine  any 
significant  difference  between  the  level  of  preparedness  and 
the  level  of  applicability  for  each  of  the  engineering 
related  skills.  For  these  tests,  the  N  value  for  the 
civilian  group  ranged  from  534  to  541,  and  the  N  value  for 
the  military  group  ranged  from  517  to  521.  The  results  are 
summarized  in  Table  18. 

All  but  three  differences  are  significant  at  the  p<  .05 
level.  A  positive  value  for  the  mean  difference  indicates 
that  respondents,  on  average,  gave  higher  ratings  to  their 
perceived  level  of  preparedness  relative  to  the  rating  of 
skill  applicability.  Conversely,  a  negative  value  for  the 
mean  difference  indicates  that  respondents,  on  average,  gave 
lower  ratings  to  their  perceived  level  of  preparedness 
relative  to  the  rating  of  applicability  for  that  skill. 
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Table  18:  Paired  Sample  T-Test  (Mean  Differences  for 

Civilian /Military) 


Engineering  Skill  /  Ability 

Mean  Difference 
( Pr epar edne  s  s / Appl i cabi 1 i ty ) 

CIVILIAN 

MILITARY 

Ability  to  apply  knowledge  of 
math,  engineering  and  science. 

0.47 

0.51 

Ability  to  analyze  and 
interpret  data. 

No  significant 
difference 

0.2 

Ability  to  identify,  formulate 
and  solve  engineering  problems . 

0.94 

0.98 

Ability  to  design  systems, 
components  or  processes  to  meet 
needs . 

0.27 

0.94 

Ability  to  design  and  conduct 
experiments . 

0.87 

1 . 14 

Ability  to  use  techniques, 
skills  and  tools  in  engineering 
practice . 

0.73 

0.82 

Ability  to  use  technologies 
related  to  your  career  field. 

-0.38 

-0.13 

Ability  to  prepare  reports  and 
documents . 

-0.38 

No  significant 
difference 

Ability  to  prepare  and  deliver 
professional  presentations. 

-0.58 

-0.35 

Ability  to  function  on  multi¬ 
disciplinary  teams. 

-0.29 

No  significant 
difference 

Ability  to  accept  ethical  and 
professional  responsibility. 

-0.16 

-0.21 

Importance  of  engaging  in  life¬ 
long  learning. 

-0.15 

-0.19 

For  civilian  respondents,  there  was  no  significant 
difference  between  their  level  of  applicability  and  their 
level  of  preparedness  for  their  ability  to  apply  knowledge 
of  math,  engineering  and  science. 

For  military  respondents,  there  was  no  significant 
difference  between  their  level  of  applicability  and  their 
level  of  preparedness  for  the  following  engineering  skills: 
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1. 


Ability  to  prepare  reports  and  documents. 


2.  Ability  to  function  on  multi-disciplinary  teams. 

For  all  other  paired  questions,  there  were  significant 
differences  between  the  mean  values  of  applicability  and 
preparedness,  for  both  military  and  civilian  respondents. 

For  the  skills  listed  below,  the  difference  between  the 
level  of  preparedness  and  the  level  of  applicability  was 
positive,  meaning  that  respondents  (on  average)  gave  high 
ratings  for  preparedness  in  these  skill  areas  relative  to 
the  level  of  skill  applicability: 

1.  Ability  to  apply  knowledge  of  math,  engineering  and 
science . 

2.  Ability  to  analyze  and  interpret  data. 

3.  Ability  to  identify,  formulate  and  solve  engineering 
problems . 

4.  Ability  to  design  systems,  components  or  processes 
to  meet  needs . 

5.  Ability  to  design  and  conduct  experiments. 

6.  Ability  to  use  techniques,  skills  and  tools  in 
engineering  practice. 

For  the  skills  listed  below,  the  difference  between  the 
level  of  preparedness  and  the  level  of  applicability  was 
negative,  meaning  that  respondents,  on  average,  gave  low 
ratings  for  preparedness  in  these  skill  areas  relative  to 
the  level  of  skill  applicability: 

1.  Expectation  to  be  current  in  technologies  related 
to  your  current  field. 
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2.  Ability  to  prepare  reports  and  documents. 

3.  Ability  to  prepare  and  deliver  professional 
presentations . 

4.  Ability  to  function  on  multi-disciplinary  teams. 

5.  Understanding  of  ethical  and  professional 

responsibility . 

6.  Recognition  of  the  need  to  engage  in  life-long 
learning . 
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V.  SUMMARY,  CONCLUSION  AND  RECOMMENDATIONS 


A.  SUMMARY 

There  were  several  purposes  to  this  research.  The 
primary  purpose  was  to  determine  the  extent  to  which  Naval 
Academy  graduates  who  majored  in  engineering  believe  their 
undergraduate  education  to  be  relevant  to  their  current 
profession,  utilizing  the  Accreditation  Board  of  Engineering 
and  Technology  (ABET)  "student  learning  outcomes"  (ABET, 
2005,  p.  1)  as  the  basis  for  assessing  relevance.  The 
intention  was  to  assess  both  the  applicability  of  these 
particular  skills  to  graduates'  current  professions,  as  well 
as  the  perceived  levels  of  preparedness  in  these  skill  areas 
afforded  respondents  due  to  their  undergraduate  education. 

A  secondary  purpose  was  to  draw  meaningful  comparisons 
between  different  groups  within  the  sample.  These  comparison 
groups  include  military  and  civilian,  technical  and  non¬ 
technical  jobs,  and  comparison  across  the  different 
engineering  majors. 

A  third  purpose  of  this  research  was  to  establish  a 
body  of  baseline  data  about  engineering  majors  who  have 
graduated  from  the  Naval  Academy  throughout  the  past  20 
years.  The  intention  was  to  gather  a  broad  base  of  data 
about  what  professions  or  jobs  graduates  are  currently  doing 
and  how  their  undergraduate  education  has  factored  into 
their  careers. 

An  overarching  goal  of  this  research  is  to  examine  the 
relationship  between  engineering  education  at  the  Naval 

Academy  and  the  impact  that  education  has  on  graduates  after 
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they  graduate  and  begin  their  professional  lives. 
Respondents  have  provided  valuable  information  and  insight 
based  on  their  personal  experience,  about  what  skills  have 
been  applicable  to  their  professional  lives  and  how  they 
perceive  their  level  of  preparation  due  to  their  engineering 
education.  The  emphasis  is  on  identifying  the  strengths  of 
the  educational  program,  as  well  as  identifying  potential 
areas  for  future  improvement. 

B.  CONCLUSION 

Overall,  Naval  Academy  graduates  who  majored  in 
engineering  are  generally  satisfied  with  the  quality  of 
their  undergraduate  education.  They  also  feel  that  what  they 
studied  in  their  engineering  program  is  applicable  to  what 
they  do  today. 

For  6  of  the  12  engineering  related  skills  or  abilities 
that  were  analyzed  in  this  research,  respondents  indicated 
very  high  levels  of  importance  to  their  current  work.  The 
six  most  important  skills  were:  "understanding  of  ethical 
and  professional  responsibility;"  "prepare  and  deliver 
professional  presentations;"  "recognition  of  the  need  to 
engage  in  life-long  learning;"  "function  on  multi¬ 
disciplinary  teams;"  and  "ability  to  analyze  and  interpret 
data."  All  of  these  skills  received  ratings  of  importance 
between  4.38  and  4.74  (i.e.,  between  "very  important"  and 
"extremely  important")  . 

The  skills  that  were  rated  as  least  important  to 
respondents  were:  "design  and  conduct  experiments;"  "design 
systems,  components  or  processes  to  meet  needs;"  "use 
techniques,  skills  and  tools  in  engineering  practice;"  and 


60 


These 


"identify,  formulate  and  solve  engineering  problems, 
skills  received  ratings  of  importance  between  2.34  and  3.09 
(i.e.,  between  "somewhat  important"  and  "fairly  important"). 
For  all  of  these  skills,  respondents  (on  average)  indicated 
high  levels  of  preparedness  relative  to  the  level  of 
applicability  for  the  particular  skill. 

In  general,  it  seems  that  respondents  gave  lower 
ratings  of  importance  to  skills  that  could  be  seen  as 
particular  to  technical  work.  Respondents  gave  higher 
ratings  of  importance  to  skills  that  are  more  universally 
applicable . 

Respondents  felt  well  prepared  for  the  abilities  to 
"design  and  conduct  experiments"  and  "identify,  formulate 
and  solve  engineering  problems,"  yet  reported  low  ratings 
for  the  applicability  of  these  skills  to  their  current  job. 

Conversely,  respondents  rated  their  preparedness  low 
relative  to  the  level  of  applicability  in  the  skill  areas  of 
"expectation  to  be  current  in  technologies  related  to 
current  career  field;"  "ability  to  prepare  and  deliver 
professional  presentations;"  and  ability  to  "prepare  written 
reports  and  documents;"  "ability  to  function  on  multi¬ 
disciplinary  teams;"  "understanding  of  ethical  and 
professional  responsibility"  and  "recognition  of  the  need  to 
engage  in  life-long  learning."  For  these  skills,  respondents 
reported  high  levels  of  preparedness,  but  they  reported 
extremely  high  levels  of  applicability,  especially  for  the 
skills  related  to  preparing  and  delivering  presentations  and 
preparing  reports  and  documents. 

As  could  be  expected,  respondents  who  indicated  that 
their  current  job  or  profession  was  technical  in  nature  felt 
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that  their  undergraduate  engineering  education  had  prepared 
them  for  their  job  to  a  significantly  higher  degree  than 
those  respondents  who  indicated  that  their  current  job  or 
profession  was  non-technical  in  nature.  This  result  is  not 
surprising,  considering  the  technical  nature  of  the 
respondents'  undergraduate  education. 

In  general,  which  of  the  engineering  fields  a 
respondent  majored  in  had  no  significant  effect  on  how  well 
prepared  they  felt  for  their  current  job  due  to  their 
undergraduate  education.  Sample  means  ranged  from  4.15 
(aeronautics)  to  3.75  (general  engineering),  with  an  overall 
average  of  3.95  (i.e.,  "very  well  prepared");  however,  none 
of  the  differences  between  major  groups  were  statistically 
significant . 

This  congruence  across  majors  may  be  a  result  of  the 
fact  that  regardless  of  which  particular  field  a  respondent 
chose  to  major  in,  all  were  required  to  take  a  high 
percentage  of  core  curriculum  courses  that  are  common  to  all 
engineering  majors.  While  each  major  offers  many  technical 
courses  that  are  specific  to  that  particular  field,  there 
are  a  large  number  of  common  courses  that  provide  all 
engineering  majors  with  similar  technical  and  engineering 
related  skills. 

Respondents  who  had  gone  on  to  either  complete  a 
master' s  degree  or  begin  working  on  a  master' s  degree  felt 
significantly  better  prepared  for  graduate  education  than 
those  respondents  who  had  not  advanced  their  formal 
education  beyond  their  bachelor's  degree.  This  result  seems 
to  indicate  that  USNA  graduates  who  majored  in  engineering 
underestimate  the  degree  to  which  they  are  prepared  to 
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pursue  advanced  education  in  the  form  of  a  master's  program. 
However,  it  should  be  noted  that  respondents  were  not  asked 
to  provide  any  specific  information  about  the  nature  of 
their  master's  program. 

1 .  Civilian  /  Military  Comparison 

On  average,  respondents  who  were  still  active  duty 
military  gave  slightly  higher  ratings  to  their  feelings  of 
preparedness  for  their  current  job  than  those  respondents 
who  have  transitioned  to  the  civilian  work  force.  The 
difference  between  military  and  civilian  was  statistically 
significant  at  the  p  <  .10  level,  but  not  at  the  p  <  .05 
level . 

This  difference  in  perceived  level  of  preparedness 
could  be  driven  in  part  by  the  fact  that  a  good  deal  of  the 
core  curriculum  offered  at  the  Naval  Academy  is  focused  on 
military  -  specific  education,  such  as  courses  in 
leadership,  tactics,  military  history  and  navigation.  These 
courses  are  designed  specifically  to  prepare  graduates  for 
the  time  they  will  serve  as  officers  in  the  Navy  or  Marine 
Corps . 

For  9  of  the  12  specified  engineering  related  skills, 
there  was  no  significant  difference  between  civilian  and 
military  respondents  with  respect  to  their  perceived  levels 
of  preparedness.  Interestingly,  for  the  three  skills  where  a 
significant  difference  did  exist,  civilians  gave  higher 
ratings  to  their  level  of  preparedness  than  the  military 
respondents.  These  skills  were  "the  ability  to  apply 
knowledge  of  math,  engineering  and  science;"  "function  on 
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multi-disciplinary  teams;"  and  "understanding  of 
professional  and  ethical  responsibility." 

With  respect  to  the  perceived  difference  between  level 
of  preparedness  and  the  level  of  applicability  of  each 
engineering  related  skill  or  ability,  there  are  several 
notable  differences  between  civilian  and  military 
respondents.  Civilian  respondents  gave  low  ratings  of 
preparedness  to  the  skills  of  "preparing  reports  and 
documents"  and  "ability  to  function  on  multi-disciplinary 
teams"  relative  to  the  level  of  these  skills'  applicability. 
Military  respondents  showed  no  significant  difference 
between  preparedness  and  applicability  with  respect  to  these 
skills . 


It 

is 

interesting  to  note  that 

although  civilian 

respondents 

reported  higher  levels 

of 

preparedness 

than 

military 

respondents  with  respect 

to 

"the  ability 

to 

function 

on 

multi-disciplinary  teams. 

"  civilians  still 

felt 

under-prepared  in  this  skill  area,  whereas  military 
respondents  did  not.  This  can  be  attributed  to  civilian 
respondents  that  reported  much  higher  levels  of 
applicability  (4.72)  than  military  respondents  (4.30)  for 
this  given  skill.  For  all  other  skills,  there  were  slight 
variations  between  civilian  and  military  respondents,  but  in 
general,  both  groups  reported  similar  results  when  comparing 
levels  of  applicability  and  preparedness. 

C.  RECOMMENDATIONS  AND  LIMITATIONS 

One  limitation  to  this  study  was  caused  by  lack  of 
comprehensive  access  to  Naval  Academy  graduates  who  majored 
in  an  engineering  field.  Only  those  graduates  who  maintain 
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current  contact  information  with  the  Naval  Academy  Alumni 
Association  were  able  to  be  contacted,  which  certainly 
degrades  the  randomness  of  the  sample. 

Another  limitation  is  that  this  research  was  conducted 
in  an  entirely  quantitative  fashion.  There  is  a  large  amount 
of  qualitative  data  that  was  not  analyzed.  These  data  are 
from  the  part  of  the  survey  instrument  that  allowed 
respondents  to  write  their  own  comments  in  response  to 
several  questions.  Respondents  were  asked  to  comment  on  the 
most  useful  aspects  of  their  engineering  education,  the 
least  useful  aspects  of  their  engineering  education,  and 
finally  to  offer  additional  comments  or  suggestions  for 
improvement  of  the  program.  The  written  answers  to  these 
questions  will  provide  a  good  starting  point  for  a 
qualitative  assessment  of  the  Naval  Academy' s  engineering 
program,  and  would  be  an  excellent  opportunity  for  future 
research . 

Overall,  the  results  of  this  analysis  demonstrate  that 
graduates  of  the  Naval  Academy  who  majored  in  engineering 
are  satisfied  with  their  undergraduate  education;  they 
believe  that  most  of  what  they  studied  is  applicable  to  what 
they  do  now;  and  that  they  were  well  prepared  in  most  areas. 
Possible  areas  for  program  improvement  would  be  to  give 
students  more  opportunities  to  learn  how  to  prepare  and 
deliver  professional  presentations  and  prepare  written 
reports  and  documents;  these  skills  will  greatly  benefit 
them  after  graduation.  Additional  opportunities  for  students 
to  participate  in  multi-disciplinary  groups,  or  to  work  on 
group  projects,  will  provide  them  with  valuable  experience 
that  will  be  useful  to  them  in  their  professional  careers. 
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APPENDIX  A: 


GRADUATE  PERFORMANCE  ASSESSMENT  SURVEY 


USNA  ABET  GRADUATE  SURVEY 
Division  of  Engineering  and  Weapons 
Administered  Jan-Feb  2006 


Survey  flow: 

1.  Everyone  takes  the  first  part  of  the  survey. 

2.  Only  currently  active  military  take  the  “active  military'  part  of  the  survey. 

3.  All  others  take  the  'non-active  military’  part  of  the  survey. 

4.  Everyone  takes  the  last  part  of  the  survey. 


The  next  page  starts  the  FIRST  PART  of  the  survey. 
(Everyone  takes  this  part  of  the  survey.) 
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USNA  ABET  Graduates  Survey 


Page  1  of  2 


This  survey  is  intended  to  collect  assessment  information  regarding  each  of  our  engineering 
programs  (majors)  at  USNA.  The  information  collected  via  this  survey  will  be  analyzed  and 
used  by  each  department  to  help  improve  their  respective  programs  in  order  to  best  prepare 
future  graduates  for  their  military  and  professional  careers. 

You  are  asked,  as  a  graduate  of  a  USNA  engineering  program,  to  provide  the  following 
information  voluntarily  and  anonymously.  No  identifying  information  will  be  linked  to  your 
responses.  Questions  regarding  the  survey  may  be  directed  toward  Associate  Professor  Jennifer 
Waters. 


What  is  your  current  military  duty  status? 


Active  Duty  (non-reservist) 

Reservist  on  Active  Duty  (>=180  days.  Recall  or  Mobilized) 

Reservist  on  Active  Duty  (<180  days,  AT,  ADT,  or  ADSW) 

Drilling  Reservist  (SELRES  or  IRR) 

Retired  (Active  or  Reserve) 

Civilian 

How  many  years  were  you  or  have  you  been  in  active  duty  status  since 
graduating  from  USNA? 

Select  #  of  Years 


Since  graduating  from  USNA,  how  many  years  (if  any)  have  you  been  in  a  non¬ 
active  duty  status? 

Select  #  of  Years 

4.  What  is  your  current  or  highest  military  rank  achieved? 


llo-i  ! 

ii  i 

http://www.usna.edu/IR/surveys/abet-grads/stepl  .php 


1/24/2006 
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o 

0-2 

0-3 

0-4 

0-5 

0-6 

flag  Rank 

|  Continue  ) 


http ://  www.  usna.  e  du/IR/ surveys/abet-grads/ st  ep  1  .php 


1/24/2006 
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The  next  page  starts  the  ACTIVE  MILITARY  PART  of  the  survey. 
(Active  military  only  take  this  part  of  the  survey.) 
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ABET  -  USNA  Graduate  Performance  Assessment  Survey 


Page  1  of  2 


Graduate  Performance 
Assessment  Survey 

Phase  I:  USNA  Engineering  Majors  Graduates 


In  which  community  do  you  serve? 


Aviation 

Spec  Ops 

Spec  Warfare 

Subs 

Surface 

Restricted  Line  /  Staff  Corps:  AEDO 

Restricted  Line  /  Staff  Corps:  AMDO 

Restricted  Line  /  Staff  Corps:  Chaplain  Corps 

Restricted  Line  /  Staff  Corps:  Civil  Engineer  Corps 

Restricted  Line  /  Staff  Corps:  Engineering  Duty 

Restricted  Line  /  Staff  Corps:  Human  Resources 

Restricted  Line  /  Staff  Corps:  Information  Professional 

Restricted  Line  /  Staff  Corps:  Information  Warfare 

Restricted  Line  /  Staff  Corps:  Intelligence 

Restricted  Line  /  Staff  Corps:  JAG  Corps 

Restricted  Line  /  Staff  Corps:  Medical  Assignments 

Restricted  Line  /  Staff  Corps:  METOC 

Restricted  Line  /  Staff  Corps:  Public  Affairs 

Restricted  Line  /  Staff  Corps:  Supply  Corps 

Marine  Air 

Marine  Ground 

Other 

Please  list  any  military  professional  schools  you  have  attended  (e.g.,  flight 


http://www.usna.edu/IR/surveys/abet-grads/step2.php 
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ABET  -  USNA  Graduate  Performance  Assessment  Survey 


Page  2  of  2 


school,  nuclear  power  school,  TBS,  etc.): 


Would  you  assess  the  nature  of  your  current  position  as  technical  or  non¬ 
technical? 


Technical 

Non-technical 

Do  you  intend  to  complete  a  career  serving  in  the  military  ? 


Definitely  Yes 

Undecided 

Definitely  No 

Continue 


http://www.usna.edu/IR/surveys/abet-grads/step2.php 


1/24/2006 
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The  next  page  starts  the  NON-ACTIVE  MILITARY  PART  of  the  survey. 
(Non- active  military  only  take  this  part  of  the  survey.) 
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Graduate  Performance 
Assessment  Survey 


Phase  I:  USNA  Engineering  Majors  Graduates 


Please  answer  the  following  questions  regarding  your  CURRENT  PROFESSIONAL 
(CIVILIAN)  EMPLOYMENT: 

What  is  your  employment  status? 


Currently  employed  full-time 

Currently  employed  part-time 

Self-employed 

Not  currently  employed,  and  looking 

Not  currently  employed,  and  not  looking 

Retired  full-time 

If  you  are  currently  employed  full  or  part-time,  please  complete  the  following: 

Position  Title: 

Name  of  Employer: 

City/State: 


Which  of  the  following  choices  best  describes  your  employment  and  employer? 


Engineering/T echnical  position  in  an  engineering/technically-based  organization 

Non-technical  position  in  a  technically-based  organization 

Te  clinical  position  in  a  non-technically  based  organization 

Non-technical  position  in  a  non- technically-based  organization 

Which  of  the  following  levels  best  describes  your  current  position? 

i  — i 


http://www.usna.edu/IR/surveys/abet-grads/step2.php 
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Senior  Executive  (President.  CEO.  COO.  etc.) 

tipper  Management  (Vice  President.  Director,  etc.) 

Senior  Technical/Professional 

Mid-Level  Manager 

Mid-Level  Technical/Professional 

Entry  Level 

* 

Other 

Is  your  current  position  related  to  your  major  at  USNA? 


1 

To  a  great  extent 

I 

Somewhat 

1 

Not  at  all 

How  many  years  have  you  held  your  current  position? 

Select  #  of  Years 

[  Continue  J 


http://www.usna.edu/IR/surveys/abet-grads/step2.php 
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The  next  page  starts  the  LAST  PART  of  the  survey. 
(Everyone  takes  this  part  of  the  survey.) 


76 


ABET  -  USNA  Graduate  Performance  Assessment  Survey 


Page  1  of  5 


Graduate  Performance  Assessment 
Survey 

Phase  I:  USNA  Engineering  Majors  Graduates 


Are  you  willing  to  be  listed  in  a  database  for  midshipmen  to  contact  regarding  caree 
or  other  professional  information? 


No 

Yes  -  Please  give  e-mail  address  or  other  contact  info  below.  Alternately,  if  you  prefer,  youma> 
send  an  e-mail  to:  Associate  Professor  Jennifer  Waters.  iwaters®.usna.edu. 

Enter  your  contact  information  here: 

Upon  completion  of  your  engineering  majors  program  at  USNA,  rate  your  preparation  for  the 
following  abilities/characteristics: 


Not  at  all 
Prepared 

Somewhat 

Prepared 

Adequate 

Very 

Prepared 

Extreme 

Prepare 

Ability  to  apply  knowledge  of  math, 
engineering,  and  science. 

o 

o 

o 

o 

o 

Ability  to  analyze  and  interpret  data. 

Ability  to  identity,  formulate,  and  solve 
engineering  problems. 

o 

o 

o 

o 

o 

Ability  to  design  system,  component,  or 
process  to  meet  needs. 

o 

o 

o 

o 

o 

Ability  to  design  and  conduct 
experiments. 

o 

o 

o 

o 

o 

Ability  to  use  techniques,  skills,  and  tools 
in  engineering  practice. 

o 

o 

o 

o 

o 

Expectation  to  be  current  in  technologies 
related  to  your  career  field. 

o 

o 

o 

o 

o 

Ability  to  prepare  reports  and 
documents. 

o 

o 

o 

o 

o 

http://www.usna.edu/IR/surveys/abet-grads/step3.php 


1/24/2006 


77 


ABET  -  USNA  Graduate  Performance  Assessment  Survey 


Page  2  of  5 


Ability  to  prepare  and  deliver 
professional  presentations. 

O 

o 

° 

° 

o 

Ability  to  function  on  multi-disciplinary 
teams. 

O 

o 

o 

o 

o 

Understanding  of  ethical  and 
professional  responsibility. 

o 

o 

° 

o 

o 

Recognition  of  need  to  engage  in  life-long 
learning. 

° 

o 

o 

o 

o 

In  your  current  position,  how  important  are  the  following? 


Not  at  all 
Important 

Soiihm  hat 
Important 

Fairly 

Important 

Very 

Important 

Extremely 

Important 

Not 

Applicab 

To  apply  knowledge  of 
math,  engineering,  and 
science. 

° 

° 

o 

o 

o 

o 

To  analyze  and  interpret 
data  to  solve  open-ended 
problems. 

o 

o 

o 

o 

o 

o 

To  identify,  formulate,  and 
solve  engineering 
problems. 

o 

o 

o 

o 

o 

o 

To  design  system, 
component,  or  process  to 
meet  needs. 

o 

o 

o 

o 

o 

o 

To  design  and  conduct 
experiments. 

o 

o 

o 

o 

o 

o 

To  use  techniques,  skills, 
and  tools  in  engineering 
practice. 

o 

o 

o 

o 

o 

o 

To  be  current  in 
technologies  related  to 
your  career  field. 

o 

o 

o 

o 

o 

o 

To  prepare  reports  and 
documents. 

o 

o 

o 

o 

o 

o 

To  prepare  and  deliver 
professional  presentations. 

o 

o 

o 

o 

o 

o 

To  function  on  multi¬ 
disciplinary  teams. 

o 

o 

o 

o 

o 

o 

Ethical  and  professional 
responsibility. 

o 

o 

o 

o 

o 

o 

To  engage  in  life-long 

1  1 

http://www.usna.edu/IR/surveys/abet-grads/step3.php 
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I1”™8"*  I  II  II  I  I  I 


How  well  did  your  major  at  USNA  prepare  you  for  your  current  career? 


Not  at  all  Prepared 

Somewhat  Prepared 

Adequate 

Very  Prepared 

Extremely  Prepared 

How  often  do  you  attend  professional  conferences,  meetings,  symposia,  etc.? 


ce  per  year 


IjBctwecn  two  (2)  and  five  (5)  times  per  year 


||More  th«m  five  (5)  times  per  year 


Do  you  hold  a  professional  engineer's  (P.E.)  license? 


Yes 

I  have  passed  the  FE  or  EIT  exam,  and  intend  to  pursue  P.E.  licensing 

I  have  passed  the  FE  or  EIT  exam,  but  do  not  intend  to  pursue  P.E.  licensing 

No.  but  I  intend  to  pursue  P.E.  licensing 

No  and  do  not  intend  to  pursue  P.E.  licensing 

List  the  professional  associations  or  societies  to  which  you  belong: 


List  any  significant  technical  honors,  aw  ards,  or  patents  received: 


http://www.usna.edu/lR/surveys/abet-grads/step3.php 
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Whether  or  not  you  have  attended  graduate  school,  how  well  do  you  believe  your 
majors  program  at  USNA  prepared  you  for  graduate  school? 


Not  at  all  Prepared 

Somewhat  Prepared 

Adequate 

Very  Prepared 

Extremely  Well  Prepared 

Indicate  the  highest  degree  you  have  attained: 


Please  specify  each  additional  degree  -  including  discipline  -  you  have  attained  since 
graduating  from  USNA  and  the  school(s)  attended.  For  example:  Ph.D.,  Mechanical 
Engineering,  University  of  Maryland;  M.S.,  Systems  Engineering,  Naval 
Postgraduate  School. 


Briefly  list  the  most  useful  aspect(s)  of  your  engineering  majors  program  at  USNA: 


http://www.usna.edu/IR/surveys/abet-grads/step3.php 
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Briefly  list  the  least  useful  aspect(s)  of your  engineering  majors  program  at  USNA : 


Additional  comments  or  suggestions  for  improvement  regard  mg,  your  engineering 
majors  program  at  USNA: 


Please  describe  any  experiences  or  feedback  related  to  your  academic  program  not 
adequately  captured  elsewhere  in  this  surv  ey. 


[  Submit  Information  ] 


http://www.usna.edu/IR/surveys/abet-grads/step3.php 
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APPENDIX  B:  SAMPLE  DESCRIPTIVE  STATISTICS 


Table  19:  Highest  Military  Rank  Achieved 


Rank 

Frequency 

Percent 

0-1 

87 

8 . 1 

0-2 

107 

10.0 

0-3 

493 

46.2 

0-4 

268 

25.1 

0-5 

108 

10.1 

0-6 

1 

0.1 

Missing 

4 

0.4 

Total 

1068 

100.0 

Table  20:  Warfare  Community 


Community 

Frequency 

Percent 

Aviation 

224 

43.5 

Submarine 

Warfare 

79 

15.3 

Surface 

Warfare 

48 

9.3 

USMC 

69 

13.4 

SPECWAR  / 
SPECOPS 

9 

1.7 

Civ .  Eng . 

Corps  / 
Eng .  Duty 
Officer 

46 

8 . 9 

Other 

40 

7 . 8 

83 


Table  21:  Job  Description  (Military) 


Job  Description 

Frequency 

Percent 

Technical 

357 

69.3 

Non-Technical 

158 

30.7 

Total 

515 

100.0 

Table  22:  Job  Description  (Civilian) 


Job  Description 

Frequency 

Percent 

Engineering  / 
Technical  Position 
in  Engineering  / 
Technically  based 
organization 

259 

48.0 

Non- technical 
position  in 
Technically  based 
org. 

107 

19.8 

Technical  position 
in  Non-Tech.  org. 

49 

9.1 

Non-tech  position 
in  Non- tech  org. 

125 

23.1 

Total 

540 

100.0 
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Table  23:  Job  Level  (Civilian) 


Level  within 
Organization 

Frequency 

Percent  and 
Valid  Percent 

Cumulative 

Percent 

Senior  Exec. 
(President, 
CEO,  COO, 
CIO,  CFO) 

40 

7.4 

7.4 

Upper  Mngmt 
(VP,  ~ 
Director) 

101 

18.7 

26.1 

Senior 
technical  / 
Professional 

118 

21 . 8 

47 . 9 

Mid- level 
Management 

130 

24.0 

71 . 9 

Mid-level 
technical  / 
professional 

107 

19.8 

91.7 

Entry  Level 

18 

3.3 

95.0 

Other 

27 

5.0 

100.0 

Total 

541 

100.0 
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